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FOREWORD 


These proceedings contain the papers of the 11 th International Conference on Cognition 
and Exploratory Learning in the Digital Age (CELDA 2014), 25-27 October 2014, 
which has been organized by the International Association for Development of the 
Information Society (IADIS) and endorsed by the Japanese Society for Information and 
Systems in Education (JSISE). 

The CELDA 2014 conference aims to address the main issues concerned with evolving 
learning processes and supporting pedagogies and applications in the digital age. There 
have been advances in both cognitive psychology and computing that have affected the 
educational arena. The convergence of these two disciplines is increasing at a fast pace 
and affecting academia and professional practice in many ways. 

Paradigms such as just-in-time learning, constructivism, student-centered learning and 
collaborative approaches have emerged and are being supported by technological 
advancements such as simulations, virtual reality and multi-agents systems. These 
developments have created both opportunities and areas of serious concerns. This 
conference aims to cover both technological as well as pedagogical issues related to 
these developments. Main tracks have been identified. However innovative 
contributions that do not easily fit into these areas will also be considered as long as 
they are directly related to the overall theme of the conference - cognition and 
exploratory learning in the digital age. 


The following areas are represented in the submissions for CELDA 2014: 


• Acquisition of expertise 

• Assessing progress of learning in 
complex domains 

• Assessment of exploratory learning 
approaches 

• Assessment of exploratory 
technologies 

• Cognition in education 

• Collaborative learning 

• Educational psychology 

• Exploratory technologies 
(simulations, VR, i-TV, etc.) 

• Just-in-time and Leaming-on- 
Demand 


• Learner communities and peer- 
support 

• Learning communities & Web service 
technologies Pedagogical issues 
related with learning objects 

• Learning paradigms in academia 

• Learning paradigms in the corporate 
sector 

• Life-long learning 

• Student-centered learning 

• Technology and mental models 

• Technology 

• Learning and expertise 

• Virtual university 


The CELDA 2014 Conference received 78 submissions from more than 20 countries. 
Each submission was reviewed in a double -blind review process by at least two 
independent reviewers to ensure quality and maintain high standards. Out of the papers 
submitted, 25 were accepted as full papers for an acceptance rate of 32%; 17 were 
accepted as short papers and 2 were accepted as reflection papers. Authors of the best 
published papers in the CELDA 2014 proceedings will be invited to publish extended 
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versions of their papers in a special issue of the International Journal Technology, 
Knowledge and Learning and in a book from Springer. 

In addition to the presentation of full papers, short papers and reflection papers, the 
conference also includes a keynote presentation from an internationally distinguished 
researcher. We would therefore like to express our gratitude to Professor Jan Elen, 
Faculty of Psychology and Educational Sciences, K.U. Leuven, Belgium, as the 
CELDA 2014 keynote speaker. 

The conference will also include a panel entitled "Competencies, Challenges, And 
Changes: A Global Conversation About 21 st Century Teachers And Leaders" with 
Lynne Schrum, Rose Dolan, Dirk Ifenthaler, Ronghuai Huang, Dale Niederhauser and 
Neal Strudler. 

A successful conference requires the effort of many individuals. We would like to thank 
the members of the Program Committee for their hard work in reviewing and selecting 
the papers that appear in this book. We are especially grateful to the authors who 
submitted their papers to this conference and to the presenters who provided the 
substance of the meeting. We wish to thank all members of our organizing committee. 

Last but not least, we hope that participants enjoy Porto and their time with colleagues 
from all over the world. 

Pedro Isaias, Universidade Aberta (Portuguese Open University), Portugal 
Conference Chair 

Demetrios G. Sampson, University of Piraeus and CERTH, Greece 
J. Michael Spector, University of North Texas, USA 
Dirk Ifenthaler, Deakin University, Australia 
Program Co -Chairs 


Porto, Portugal 
October 2014 
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KEYNOTE LECTURE 


DESIGNING LEARNING ENVIRONMENTS IN A 
DIGITAL AGE: THE QUEST FOR WHAT TO CONSIDER 


Professor Jan Elen, 

Faculty of Psychology and Educational Sciences, 
K.U. Leuven, 

Belgium 


Abstract 

Designing effective and efficient learning environments remains a great challenge. In 
order to do so, we build on the rich instructional design tradition that proposes the 
variables to consider. That rich tradition is rooted in a time in which learning 
environments were non-digital. Interesting work was done on how to design a ‘lesson’ 
and at least as much research was done on the design of printed materials. While 
‘lessons’ are still delivered and printed materials remain important, the presence of 
mostly digital environments has challenged a number of the ideas prevalent in the 
instructional design tradition. This challenge is not in as much the result of a change of 
medium as it is the result of changes that have come along with the arrival of digital 
learning environments: changes with respect to what is to be learned in a digital age; 
evolutions in our understanding of learning processes; an ever increasing number of 
(digital) tools to support learning. 

In this contribution we will review some of these changes in view of specifying 
variables to consider in the design of learning environments in a digital age. 
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PANEL 


COMPETENCIES, CHALLENGES, AND CHANGES: A GLOBAL 
CONVERSATION ABOUT 21ST CENTURY TEACHERS AND 

LEADERS 


By Lynne Schrum, Rose Dolan, Dirk Ifenthaler, Ronghuai Huang, 
Dale Niederhauser and Neal Strudler 


Abstract 

This panel discussion brings together teacher educators from different regions of the 
world; all have knowledge of the issues, regulations, challenges, and goals of preparing 
educators for the schools our students need and our world needs. The goal of the 
discussion is to engage the audience in a meaningful discussion regarding their 
experiences, and to emerge with greater understanding, best practices, and future steps 
in our endeavors. 
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COMPETENCIES, CHALLENGES, AND CHANGES: 
A GLOBAL CONVERSATION ABOUT 21 st CENTURY 
TEACHERS AND LEADERS 


Lynne Schrum 1 , Rose Dolan 2 , Dirk Ifenthaler 3 , Ronghuai Huang 4 , 

Dale Niederhauser 1 and Neal Strudler 5 

12 3 4 

West Virginia University - ” National University of Ireland - DeakinUniveresity - Beijing Normal University - 

5 University of Nevada, Las Vegas 


ABSTRACT 

This panel discussion brings together teacher educators from different regions of the world; all have knowledge of the 
issues, regulations, challenges, and goals of preparing educators for the schools our students need and our world needs. 
The goal of the discussion is to engage the audience in a meaningful discussion regarding their experiences, and to 
emerge with greater understanding, best practices, and future steps in our endeavors. 

KEYWORDS 

ICT, teacher preparation, school leadership, 21 st century schools 


INTRODUCTION 

Around the world educators, policy makers, and others are seeking best practices to prepare educators and 
leaders to improve student learning, prepare learners for their futures, increase student engagement, and 
integrate learning technologies into their curriculum; a universal goal is to ensure that all learners reach their 
full potential. Questions have been raised regarding the value of practicum, appropriate curriculum, and role 
of technology in this preparation. And yet, Fullan (2013) stated, “It is now time for technology to join the 
fray in a more purposeful way in order to transform learning for educators and learners in the 21 st century” 
(p. 3). The discovery and sharing of this “purposeful'’ way will require many educators to work together, 
share lessons learned, and invest energy in promoting policies to bring about changes (Schrum & Levin, 
2015). 

This panel brings together teacher educators from several countries; panelists are familiar with the current 
and future plans for their educator preparation programs and the questions each country is addressing. What 
do teachers need to know and be able to do today as well as tomorrow’s learners? What are the best ways to 
prepare them? How are digital technologies integrated into the entire preparation program seamlessly? What 
is the status, for example, of MakerSpaces or BYOD throughout the world? How are countries preparing for 
the educational environment needed in the next several decades? 

This interactive session will raise questions about different countries’ vision for the next generation of 
learners, a variety of approaches to the preparation of teachers and educational leaders, and create a genuine 
dialogue among the panel members and the attendees regarding all our students’ future. 

Germany: Competence in Innovation and School Development 

The standards for teacher education in Germany describe the competence for innovation and school 
development as one of four principles in pre-service teacher education complementing teaching, educating, 
and assessing (KMK, 2004). Teachers are expected to play an integral part in the process of organizational 
change by developing new forms of cooperation, implementing new learning and teaching methods, as well 
as participate in further education. Accordingly, coping professionally with school innovations has become a 
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requisite for teachers (Hsiao, Chang, & Chen, 2013). From an organizational point of view, school 
development includes three dimensions: Teaching, staff development, and organizational development 
(Rolff, 1995). It is generally accepted that schools are embedded in a multilayered organizational system 
consisting of the general educational context (e.g., a countries’ or states’ specific administration and 
curriculum), school related (internal) organizational aspects (e.g., school management, innovation), and 
classroom management. The latter is considered as the most relevant for pre-service teachers (Latz, 1992; 
Slider, Noell, & Williams, 2006; Stephenson & O’Neill, 2012; van Tartwijka, den Brokb, Veldmana, & 
Wubbelsc, 2009; Woodcock & Reupert, 2010). The argument is built around a study focussing on the 
dimension of school organization and development by exploring hovvvV = 1,004 pre-service teachers assess 
their knowledge in relevant fields and whether there are differences among pre-service teachers who 
completed their school-based training. The main finding of this study is that pre-service teachers’ perceived 
knowledge regarding SOD is rather low. Not even a six month school-based training provides deeper 
understanding of the complexity and interrelatedness of factors influencing school organization and 
development. Therefore, a competency-based training program focussing on school organization and 
development is suggested. The training program will include a game -based learning environment for 
facilitating a deeper understanding of school organization and development and further developing necessary 
competencies. As research in teacher education suggests (Ertmer, 2005; Kim, Kim, Lee, Spector, & 
DeMeester, 2013), providing pre-service teachers with practice opportunities, offering competency-based 
support, and demonstrations of example cases, both through face-to-face and virtual means, can enable them 
to better understand the complexity of school organization and development and become more successful 
educators. 

United States Perspective 

Despite 35 years of claims that technology will transform US classrooms (cf., Papert, 1980; Sheingold, 1991; 
Skinner, 1984), and massive financial investments, with estimates that overall instructional technology 
spending in US schools will likely top $56 billion in 2012 (Nagel, 2008), widespread well -integrated use of 
instructional technologies remains unrealized — underutilized by teachers and students alike 
(Ertmer&Ottenbreit-Leftwich, 2010). Perhaps the most compelling rationale for this shortcoming is a marked 
lack of resources targeted at preparing teachers to use the technologies that are installed in their classrooms 
(Niederhauser&Stoddart, 2001; Office of Technology Assessment, 1988, 1994; Project Tomorrow, 2008). 
Current 1-1 device initiatives (whether laptops, tablets, smart-phones, or other) will likely compound this 
problem as teachers, who had previously been charged with taking their students down to the computer lab 
once a week, or bringing a laptop cart into the classroom for the occasional project, now have access that 
allows them to use technology with their students all day every day. Unfortunately, professional development 
for one-to-one initiatives often involves only a brief workshop on how to use the device (typically by the 
vendor who received the district contract for the devices purchased), with little or no explicit training or 
support for meaningful, well -integrated use of technology with their students. As long as we continue to 
focus on simply installing technology in schools, without preparing and supporting teachers to use it 
effectively with their students, it seems unlikely that we will ever realize the potential of technology to help 
us reach our transformational goals. 

A Chinese Perspective 

With the development of information and communication technology (ICT) and increasing use of ICT in 
education, the education system has changed a lot. First, today’s students have always been called “Digital 
native” who were grown up with digital technology and Internet (Prensky, 2001). Researchers believed that 
digital natives preferred active rather than passive learning, preferred using digital technologies and 
collaborating to finish work (Howe & Strauss, 2003; Palfrey & Gasser, 2008; Prensky, 2010; Tapscott, 
2009). Second, in respond to students’ preference of learning in an environment infused with technology, the 
research of learning space is rising to investigate the design and evaluation of technology-rich learning 
environments, including both formal and informal learning space (Oblinger, 2006). Third, the knowledge 
content that student should learn is changing, not only for educating the 21 st century skills but also for the 
basic survival skills in information age. Therefore, teachers face great challenges in this era, and have to 
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extend their knowledge from PCK to TPACK (Jang & Chen, 2010). In realizing these challenges, MOE 
China initiated the project of improving the K-12 teacher’s ability of using ICT for education in October 
2013. The aim of the project is to build the national standard of teacher’s ability of using ICT in education, to 
train 10,000,000 teachers by the end of 2017, to evaluate teacher’s ability of using ICT in education, and to 
form the mechanism of encouraging teacher’s actively using ICT to education. In May 2014, MOE China 
released the “The standard of teacher's ability of applying information technology in primary and secondary 
schools (Trial),” including two dimensions of optimizing classroom teaching by ICT and changing learning 
approaches by ICT. In each dimension, some items from five categories of technology literacy, planning and 
preparation, organization and management, evaluation and diagnose, learning and development set national 
standards for teachers. In the information age, policy makers, researchers and practitioners should 
acknowledge the challenges and provide effective strategies for teachers to conquer these challenges. 

The Irish Perspective 

In 1998, the Education Act passed into Irish law. This was a significant moment in the Irish education system 
as it was the first education act to be passed since the formation of the Irish Republic. It arose from 
significant public consultation following the publishing of the Green Paper (1992) and the National 
Education Convention (1993) and has provided, for the first time, a statutory framework for the Irish 
Education system, clarifying the roles and responsibilities of the various stakeholders in the system. The 
legislation has also paved the way for the establishment of the Teaching Council (2006), the statutory body 
responsible for the regulation of the teaching profession and the promotion of professional standards in 
teaching. These changes have had a significant impact on the system, on the lives of principals, teachers and 
pupils and have also impacted on those responsible for the education of teachers, namely the colleges of 
education and the universities. The changes were implemented initially during the ‘Celtic tiger era’ and then 
during a period of extreme austerity within the country. At a societal level, an economic level and an 
educational level, there has been much that has challenged the system within Ireland and its understanding of 
the role of education and of the competences needed by practitioners within the system. 

Standardized Assessments and 21st Century Skills: and the Beat Goes On 

As George Bernard Shaw reminded us, “Reformers have the idea that change can be achieved by brute 
sanity.” In terms of educational policies, what would constitute that illusive sanity? As an idealistic educator 
in the 1970s, I learned to teach in the Teacher Corps, a federal program designed to prepare innovative inner 
city teachers. We learned to individualize instruction and help engage and motivate young learners. Many 
years have since passed and today’s digital technologies offer us almost unlimited opportunities to enhance 
those efforts. That said, however, our quest to achieve the ideals of the Teacher Corps have been uneven at 
best. As we’ve been pushed by federal policy to leave no child behind, we’ve ironically created another 
achievement gap — the gap between what is emphasized in standardized assessments and the skills most 
needed for the 21 st Century. Progressive educators who have embraced the potential of technology to enhance 
teaching and learning have had to justify their approaches within the unrelenting press for student 
achievement. So going forward, how will all of this play out? It appears that with the adoption of new core 
standards for learning and the development of more thoughtful assessments, there is some hope on the 
horizon. Resolving this tension, however, between the goals of innovative, 21 5,1 Century teaching and learning 
and the press for assessment and accountabilityremains a core challenge for educational reformers and those 
involved in our global conversation. 
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ABSTRACT 

People with hearing disability find it difficult to access to infonnation and communication in public places. According to 
this fact, it is considered the possibility to design a communication system based on the Spanish Sign Language (SSL), 
which helps to overcome this barrier in public environments of wide concurrence, where much of the infonnation is 
provided to users through oral communication. Therefore, we want to provide a channel of communication that will help 
to improve the integration and independence of this group and contribute to the Universal Accessibility referred in the 
International Convention on the Rights of Persons with Disabilities. 

With this objective to alleviate the problem of communication and facilitate access to infonnation, we propose an 
alternative communication system to “Public Address System, PA ” used in the public areas of wide concurrence, based 
on the translation of oral messages to the SSL, using interpreter a lively 2-dimensional (2D) character. 

For the development of this research, an airport has been chosen, due to the large volume of users that travel daily, 
availability of audiovisual infrastructure and facility to capture messages to translate. 

The videos obtained in 2D, for each message translated into SSL, have been subjected to a validation process by 
members of an association of hearing disabled, who have contributed their opinions and suggestions for improvement, 
which have been applied to the final multimedia application. 


KEYWORDS 

Hearing disability, sign language, interactive application, animation, rotoscoping. 


1. INTRODUCTION 

Within the group of people with hearing disability, many of them use as the first, and in some cases as the 
sole mode of communication, sign language. The purpose of this work is to facilitate access to information in 
environments where audible warning (warning messages, including emergency messages), is essential for 
orientation and adaptation. The intention is to contribute to the independence and integration of these people 
into society, improving their quality of life and compliance with the "Universal Accessibility for people with 
disabilities." 

In the case of persons suffering from hearing disability, it is necessary to place at their disposal all the 
available resources for access to information, "the knowledge and autonomous learning, without which the 
information technology and communication, which can be an opportunity, become a new barrier. "(Utray 
Delgado & Ruiz, 2011). Example of this is at airports, where messages considered of general interest to 
users, do not reach everyone equally. Some people with hearing disability find it difficult and, in many cases 
impossible, to receive those messages. In this sense, in recent years there have been many improvements at 
airports, introducing specialized personnel at points of information or installing induction loops to provide a 
much clearer audible information for everyone using hearing aid or cochlear implant. However, these 
interventions are not completely solving the communication problems encountered by people using only sign 
language as a communication medium. 
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As is well known, "there is no universal sign language" (Ortiz, 2005). Each country has its own sign 
language. Even within each Spanish community there are versions of the language. Due to this circumstance, 
to carry out the application posed, the Spanish sign language has been chosen, declared as official language 
in Law 27/2007 of 23 October (BOE. 255), by which Spanish sign language are recognized and regulate the 
support means for oral communication of deaf people, with hearing disability and deaf blind. Variations 
between the sign languages of the different Spanish regions are not very pronounced, except in Catalonia. 

The public places that may be subject to an action such as that posed include, among others: airports, 
ports, train stations and subway, all kinds of sports facilities, shopping centers and other public facility where 
information is transmitted to users through voice communication. In these places, the information provided 
by PA system, screens and posters may be insufficient to enable such people to cope with complete 
independence as, in addition to not listen to the messages offered by PA system, may have difficulty reading 
written texts. However, it is possible that they can recognize the messages transmitted in sign language. 

Once the various possible scenarios are established, an airport has been chosen for the development of 
this application, due to the ease of capturing messages of PA system, high user concurrency and disposition 
of large spaces where screens can be placed for the issuance of the videos in sign language. 

There are other means of disseminating information, which are increasingly in demand such as interactive 
screens by mobile applications or tablet, via WI-FI. 


2. OBJECTIVES 

The objective of this work consist to make a multimedia application to translate messages advertisements and 
notices issued by PA system and information boards to SSL, through an animated character or avatar, in two 
dimensions (2D). This intervention will allow deaf people to access information with greater autonomy. 

The issuance of these animations can be made by two ways. On the one hand, an unidirectional diffusion 
consisting of storing animations on a server and every so often projected on the screens located on the 
premises for this service, while messages are sent over the PA system. On the other hand, an interactive 
application that allows access to information on flights status in terminals, distributed throughout the 
enclosure. In addition, these devices will emit videos with the information disseminated by PA system 
sequentially to a screensaver, and can be interrupted at one point, said automatic issuance by a user to query 
information about the status of a flight of interest. 

Consequently, this intervention is to take a further step towards the integration of persons with hearing 
disability, trying regardless of the degree of it, either partially or totally, that they can move into a better 
environment adapted to disability, following the signs and messages presented through projections of short 
videos in sign language accompanied by subtitles. 


3. METHODOLOGY 

The performance of this audiovisual application is structured in different phases, which have direct and 
sequential relationship between them: capture of messages, message translation to sign language, character 
design and conversion of video messages captured in 2D animation, implementation of an interactive 
graphical environment for the selection of messages and validation of videos. 

3.1 Capture of Messages 

The types of hearing messages sent in these facilities are diverse. It can be heard recommendations on the use 
of facilities, notices or rules established in the enclosure, in addition to information on the status of flights. 
Information regarding the status of flights can also be found on monitors distributed throughout the 
enclosure. 
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In Spanish airports, managed by the company AENA, messaging is standardized. In this work has been 
chosen Gran Canaria airport for the selection and recording of audio and visual messages. The uptake of the 
messages has been collected in two different ways: Through audio recordings on the airport enclosure and 
through written information posters distributed by the airport. 

3.2 Translation of Messages to Sign Language 

The next step for the realization of the application was to translate the advertisement and notices and / or 
written to the SSL. To carry out this phase of the work it is necessary to have a SSL interpreter responsible 
for this task. 

In the flight number selection messages, included in the notices messages section, small changes will be 
made by reducing the number of digits from six to three, to get a better operability. This allows a more agile 
and entertaining presentation of the resulting application. 

3.3 Character Design and Conversion of Video Messages Captured in 2D 
Animation 

Currently there are many programs for creating vector animations very easily, offering all kinds of tools from 
the most basic to others that require more skill to use, some require licensing fees, while others are 
completely free, although these generally do not have the same benefits. Depending on the level of animation 
you want to achieve, it should be choosen the one that best suits the requirements of the final product. 

At this phase an analysis of the main tools available today was made in the field of computer animation, 
both professional and amateur. The most common vector animation programs where differences can be two 
types, free open source, like Ajax Animator, KToon, Pencil or Synfig, and the ones which have to be payed, 
such as Adobe Flash, Adobe AfterEffects, Anime Studio or Toon Boom, among others. 

After analyzing the different software options and the study of their characteristics, it was assessed what 
was more adapted to the needs and defendants technical requirements, choosing the specific animation 
software known as Adobe Flash CS5, which is an application in the form of study working on "frames" and 
is intended for the production and delivery of interactive content. It uses both vector graphics and raster 
images (raster image or bitmap), sound, program code, video streaming and bidirectional audio. Additionally, 
this program incorporates animation technique called rotoscoping. The advantage of the rotoscoping 
technique is that it replaces people’s faces, which may be more or less attractive, for an animated character. 
By using this technique, the animation vectors were obtained. With these animations and the incorporation of 
some of the elements of the scene, like the stage and wallpaper, it has been achieved the final videos in SSL, 
where the performer becomes an animated character or avatar. Using an avatar is justified by the omission of 
the identity of the performer, it "depersonalizes" him, therefore produces a neutral perception of the message. 

3.4 An Interactive Graphical Environment for the Selection of Messages 

The next phase of work involves the creation of multimedia applications, that the user can select each video 
in SSL. It aims to design a friendly environment that captures the attention of users with interactive 
applications that allow users to interact with items on the screen, such as buttons or text windows. These will 
enter the desired orders by peripherals, which allow to control such interfaces to obtain the requested 
information. The software used for the programming of such applications has been the "ActionScipt 3.0", 
which is the native programming language of Flash. 

According to the objectives, they will be performed by two multimedia applications, which are discussed 
below. 

3.4.1 Application 1: Simulation destined to Video Channel 

From the previous videos, and through appropriate programming, it will be performed a unidirectional 
diffusion, storing animations in a server and each time a message is issued by PA system, simultaneously the 
corresponding video is displayed in SSL on screens distributed on site. 
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3.4.2 Application 2: Simulation of Interactive Application to Query Flight in Electronic 
Devices 

This application has been designed an interactive query system, where the user can check the status of his 
flight by entering the code on an electronic device, which has a keyboard and monitor on which information 
is presented in SSL the state in which it is located. During the time in which no information is requested, on 
the device’s screen will be displayed a screensaver with messages in SSL made for first application. They 
will follow a set pattern of visualization, which will abort only when a user interacts with the device. At this 
point the user is asked to enter flight details he wants to view, to display the information on screen. If after a 
set time it is not interacted with the device, it returns to the screensaver. 

3.5 Validation of the Videos 

After obtaining a first version of the product, it has been subjected to a validation process as a useful tool of 
communication for people with hearing disability. For this, it has been designed a questionnaire with several 
sections that assess technical, aesthetic and intelligibility aspects in communication in SSL. In addition, a 
final section in which an assessment of the application, suggestions and proposals for improvement is sought 
is included. 

Among the videos to validate, four advertisements and four notices messages were chosen. In these 
videos, subtitles and audio were deleted, not to offer any clues about those messages to those who could read 
texts and / or have auditory perception. 

For validation of the videos on SSL, we have been in contact with the Deaf People Association of Gran 
Canaria, who supported us to conduct this phase of the work. From their suggestions, in order to get as many 
samples as possible and have the support of a sign language interpreter to help us communicate with the 
audience, it was made to coincide with the presentation of videos and recollection of surveys, with the 
completion of an event at the headquarters of this Association. 

For the test, it was arranged a room suitable for the purpose. At the beginning of the meeting, the 
attendees were explained the content and objectives of the development work that was to be submitted, with 
the help of interpreter of SSL. Then we proceeded to display videos related with the advertisements in a 42- 
inch LCD monitor, ready for the occasion. Then the questionnaires on these videos were passed to all 
present. Subsequently, the same process was followed with videos concerning notices and related surveys. 

In viewing the videos and conducting surveys, thirteen people participated, nine were women and four 
men, all aged between nineteen and sixty five years. 

To obtain a detailed study of the results, they were analyzed from three different aspects: the collective 
total responses, responses by sex workers and finally, a third analysis separating the responses by age. 


4. RESULTS 

From the methodology discussed, were the results obtained discussed below. 

Four advertisements and four notices messages were chosen. It was considered that this amount is 
sufficient to obtain an objective assessment of the level of acceptance of this work. 

Independent videos were recorded for each of the messages translated to SSL. These videos will be used 
later to make 2D animations to be included in interactive applications. The avatar obtained and the process of 
converting the images of each video to 2D animation is shown in Figure l.The process is repeated for the 
remaining frames of each video. 

In the development of the videos, it has been cared formal aspects like colors fund scene, the character, 
the speed of the gestures, the subtitling or symbols, among others, with the goal of capturing the user’s 
attention. 
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4° Phase 5° Phase 6° Phase 


Figure 1. Changes in a single frame, during the animation process 

The results obtained to the first application are shown in Figures 2, 3 and 4 Figure 2 shows the first screen 
for the main menu, which the user founds when accessing this application. There, has been inserted a 
background image composed of a scenario that is acquainted with the airport environment. For its 
composition, it was used the characters and the logo, taken from a photograph taken outside the enclosure, 
plus the addition of two buttons with the name of their function, which when clicked allows access to the 
submenu. 

When the folder shaped button on the top with an icon showing an exclamation mark is pressed, it 
redirects us to the scene of the advertisement menu, shown in Figure 3.1, where you can choose one of the 
four videos available by pressing the corresponding button. In this section we find six buttons, four for 
displaying the advertisements and two smaller ones, of which one has the function to move the scene of 
notices and the other one returns us to the main screen. 







Figure 2. Display MAIN 
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Figure 3. Playing a video advertisements menu Figure 4. Playing a video obtained from submenus 

When selecting the screen shown in Figure 2, the button located on the bottom of the main menu, sends 
us to the scene menu notices shown in Figure 4, where you can select messages with information regarding to 
flight status. In this section we find eight buttons, five of which have special behavior, because they contain 
submenus. 

The buttons of these submenus are responsible for selecting the combination of videos you want to 
display. We talked about combining videos and not watching a video, because at this stage you have to do a 
combination of five of them to obtain the final video to be played. This is because in each submenu you have 
to select a message: initial message, each of the digits of the flight and final message. Within each submenu 
there are different videos, as many as messages we can choose in it. In the submenu flight code, the virtual 
keyboard that opens when you click on each of the three buttons shown, all the letters of these submenus are 
associated with their corresponding video of the alphabet and numbers. 
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Once you have selected the corresponding digit on the flight, and have chosen the option of flight status, 
we will have available the "play" button to play the message indicating, in sign language, flight status with 
the typed code, such as shown in Figure 4. 

For "Application 2: Simulation of interactive application for flight query in electronic devices " has 
designed the screensaver system shown in Figure 4, which the user encounters when accessing this 
application. All scenes are formed under one animation pattern and is made of two elements. Initially it is 
displayed the back of the stage and an animation that is shown as a text that scrolls across the screen from 
right to left showing the message : "PRESS ANY KEY TO ENTER YOUR FLIGHT NUMBER." With it, it 
is intended that the user should be aware that act on the keyboard to get the information you want. When you 
finish the text displayed on the monitor, it starts playing one of the videos that have been animated before. 



Figure 5. Scenes and videos that make up the screen saver 

We have added new elements to these videos and that belong to the symbols of the folders in the first 
application. To display on this application, it was decided to change the scale and color, choosing in this case 
red. They have also been provided with a small animation, creating a flashing effect. This is to reinforce, in a 
subtle way, the information that is being offered to the user. 

When a user interacts with the device, he will be shown a screen on their requested flight code. After 
entering this code, the corresponding video on the state of the same is shown, but in the event that the flight 
number is entered wrongly, dynamic text is activated in the center of the screen with the following message: 
"THE FLIGHT NUMBER MARKED IS NOT AVAILABLE." 

For this application to work properly it is necessary that the interactive device has updated information on 
identification data and flight status, the same way that the information provided by other means such as 
information screens and PA system is renewed. It is necessary that the information in all media is 
simultaneously updated. This task can be performed automatically by connecting the device to the 
corresponding database or manually, in which an operator is responsible for updating this information. 

Once the results shown for both applications, the results of the evaluations obtained through surveys are 
presented. 

In relation to the compression of the videos, it can be seen that about half of the attendees suggests they 
can be improved. The results for questions related to the compression of the videos, are shown in Figure 6. 


ADVERTISEMENTS NOTICES 

VERY 

DISSATISFIED 




Figure 6. Level of satisfaction with the videos shown 
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Regarding the use of a character or avatar in 2D, 84% of attendees were in favor. If we analyze by gender, 
we see that every men and 76% of women liked the video presented with a 2D character as an interpreter of 
SSL. In the results we see that all ages prefer 2D animation, but in the age group of between twenty and 
thirty-five, the acceptance level decreases. Of the three members of this group, each have a different opinion 
to the use of an animated character. 

Regarding the use of subtitles in the videos, 73% of the participants are in favor of its use. In the analysis 
of the responses to this question by gender, we see that the vast majority of both sexes considered appropriate 
to include subtitles to the videos that have been shown to them. In the study based on the responses of 
different age, the same general trend of preference subtitles in the videos is perceived. However, it should be 
pointed out that 50% of the people in the age range of forty-six to sixty have failed to answer this question. 

Finally, an assessment is requested for the videos displayed on a scale of 1 to 10, obtaining the results 
shown in the following table. 


Table 1 . Overall rating of the videos shown 


GLOBAL ASSESSMENT OF VIDEOS ON SSL 

Advertisements 

Notices 

Rated Result Percentage 

Result Percentage 


1...4 

0 

0% 

0 

0% 

5 

1 

8% 

2 

15% 

6 

2 

15% 

0 

0% 

7 

1 

8% 

0 

0% 

8 

1 

8% 

2 

15% 

9 

1 

8% 

1 

8% 

10 

7 

54% 

8 

62% 

NULL 

0 

0% 

0 

0% 

SIGNS 

13 

100% 

13 

100% 


5. CONCLUSIONS 

This article has presented a multimedia product development that can extend the possibilities of 
communication in public places with people who use the Spanish sign language. This communication system 
can be particularly useful to provide people with hearing disability an avenue of further communication, 
contributing to an increase in personal autonomy. It also has the advantage that this model can be exported to 
any public setting where required, to convey information to users. 

The environment object to the simulated study was an airport. The results show that an application of this 
characteristics and minimal cost can provide a broader communication service based sector of the population 
who may have difficulty accessing messaging and sound alerts. In this regard, it is particularly interesting 
that no additional infrastructure is required to present at sites of public assembly. 

In addition, it has been shown that a similar submitted development provides the ability to offer users a 
friendly multimedia simulation, placed in strategic places for information points, with the aim of expanding 
the communication channels available to inform passengers’ flight status and other information that may be 
of interest. 

In particular, the use of the technique of rotoscoping to create the avatar appears to provide a valid and 
depersonalized approach to sign language translation using a human figure. 

Although the number of samples in the validation survey is small and it is necessary, therefore, a larger 
study, we can conclude that the majority of user perception is positive, although messages can be improved. 
In this sense, our hypothesis is corroborated, in which we proposed that it should be added subtitled messages 
to the videos, thereby achieving a more complete communication. The validation results also indicate the 
desirability of improving the character animation, especially in his body language. 

This work is a pilot test to assess the use of rotoscoping and animation technique to create a 
communicative environment improvement for the hearing disability. This model and workflow allows the 
extension of the results to the official sign languages of other countries and environments. Additionally, it 
can be used other devices that allow the reception of messages such as tablets and mobile phones. 
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The experience obtained in this work can be transferred to other fields such as education, in which has 
utility as an augmentative communication system for hearing disability people. In this sense, it may be 
interesting to incorporate a virtual interpreter in the class that repeat the oral explanation of the teacher, in 
sign language. 
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ABSTRACT 

We introduce an educational game-based assessment that measuresthe choices students make while learning. We present 
Posterlet, a game designed to assess students’ choices to seek negative feedback and to revise, in which students learn 
graphical design principles while creating posters. We validate our game-based assessment approach with three research 
studies, in whichcollege and middle-school students play Posterlet and then complete a posttest. Results showed that the 
game helped students learn: students who played Posterlet before completing the posttest learned more graphical design 
principles than students who only completed the posttest. Moreover, the choices to seek negative feedback and to revise 
can predict learning and be used as valid outcome measures for learning. We present a first-of-kind examination of 
students’ choices to seek feedback and to revise, as well as of students’ learning outcomes based on thesechoices, which 
can be used to develop and evaluate models of instruction that help students make informed learning choices. 


KEYWORDS 

Choice, assessment, game, learning, feedback, revision 


1. INTRODUCTION 

Jean Piaget stated, “The principal goal of education is to create men [and women] who are capable of doing 
new things, not simply of repeating what other generations have done...” (Elkind, 1968, p. SM80).Our 
educational goal is to prepare students for autonomy after they leave school, because the ability to self- 
govern, to learn, and to adapt in a changing world on one’s own is crucial to a fulfilling life. Two major steps 
in operationalizing our goal are a) identifying the behaviors that capture students’ potential to learn 
autonomously and b) employing tools that measure these behaviors. Researchers have identified promising 
behaviors for learning, such as 21 st -century dispositions and attitudes (e.g., tolerance for ambiguity), but their 
relevance for improving learning has not yet been demonstrated, mostly because of a lack of assessment tools 
that can measure such behaviors. Here, we hypothesize that the choices students make when presented with a 
challenge constitute important behaviors for learning. Also, we introduce choice-basedassessments that 
measure not only students’ knowledge,but even more importantly their choices about what, when, and how to 
learn. We validate these game-based assessment environments with three research studies showing that 
choices can both predict learning and be used as valid outcome measures for it. 


2. THEORETICAL FRAMEWORK 

We ground our research in the theoretical framework of constructivist assessments (Schwartz et al., 2009) 
and in choice-based assessments specifically (Schwartz and Arena, 2009; 2013). The core of our approach 
isusing choices, or learning-related behaviors, as first-order learning outcomes and measurement constructs. 
We need assessment environments that enable us to track students’ choices and learning outcomes, so that we 
can examine the impact ofchoices on learning trajectories and measure students’ potential to learn 
independently. Thus, our ideal assessment environments must satisfy the following principles. 
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2.1 Principles of Choice-Based Assessment Environments 

Typical Performance . Assessments need to capture every-day learning behaviours. Students display 
“maximal performance” during assessment (Kleheand Anderson, 2007), which may not be congruent with 
their outside-school behaviors and performance. We need environments in whichstudents feel comfortable 
displaying their typical learning behaviors, to help us evaluate their true trajectories for lifelong learning. 

Preparation for Future Learning .Assessments need to offer learning opportunities. Traditional assessments 
are retrospective, measuring students’ knowledge at the end of instruction. Thus, they offer a snapshot of a 
current state of students’ accumulated knowledge, but provide little indication regarding future learning 
trajectories. According to Vygotsky ( 1934), measuring student knowledge at the end of instruction instead of 
measuring learning processes progressively does not yield a holistic view of students’ growth. If our goal is 
to measure behaviors that may be conducive to learning, we need to be able to embed learning in assessment 
environments. Schwartz and Bransford (1998) advocated preparation for future learningi PPL) assessments, 
which afford students opportunities to learn during the evaluation. 

Choic ^ Assessments need to gather information about unforced student choices. We need to emulate the 
choice-rich environment in which students will learn after they leave school, because lifelong learning is 
based on free choices. Moreover, students must be able to safely experiment with choices before trying them 
in the real world. Some assessment environments blend learning and assessment, such as intelligent tutoring 
systems (Koedinger et al., 1997) but, in contrast to our approach that emphasizes the value of students’ 
choices,they assume a certain sequence of steps that students take while learning, giving little choice to the 
students. Assessments cannot direct or influencestudents 1 choices. For instance, in a game-based assessment, 
students should be able to level up, no matter what learning choices they make in the game. 

2.2 The Measurement Constructs 

The Posterlet game is an instance of a choice-based PFL assessment designed to measure two behaviors 
important for learning: 1 ) the choice to seek negative feedback and 2) the choice to revise. Different learning 
contexts may afford many different learning choices that can be measured. We focused on the choice to seek 
negative feedback, because negative feedback tends to be more effective for continued learning than positive 
feedback (Klugerand DeNisi, 1998). At the same time, negative feedback runs the risk of triggering an ego 
threat that leads people to shut down rather than revise (Hattie and Timperley, 2007). While attitudes towards 
feedback are important for learning, there is no evidence whether the choice to seek feedback is important. In 
previous research, students did not exercise choice regarding feedback (but see Roll et al., 
201 1 ).Revisingmayalso bean important aspect of learning: choosing negative feedback would be of little use 
if students did not act on it. Although revising may be an important behavior for learning, there is no 
evidence in prior research whether the choice to revise is important. Thus, we investigate the impact of the 
choicestoseek negative feedbackandto revise on learning. 


3. THE ASSESSMENT ENVIRONMENT: POSTERLET 

In Posterlet) Figure 1), players design posters for different booths at a funfair. On each level, after they 
complete their initial poster design, players choose three characters to provide positive feedback (e.g., “Your 
poster has big letters. Really easy to read.”) or negative feedback(e.g., “People need to be able to read it. 
Some of your words are too small.”), which carry equivalent information. Then, players choose whether to 
revise or submit their poster. There is one round of feedback and revision for each poster. The game has three 
rounds, with nine feedback and three revision choices. Finally, the game displays students’ poster score as the 
number of tickets sold at each booth. The game’s graphical analysis system tracks the posters for the use of 
21 graphical design principles and generates poster-specific feedback. The principles governing choice-based 
assessments (typical performance. PFL, and choice) are naturally built into the game mechanics. Thegame 
provides an environment that encourages students’ typical behaviors, offering themopportunities to learn 
graphical design principles while they are exercising thcvcfeedbcick and revision choices. 
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4. CONSTRUCT VALIDATION 


1 ) Choose Booth 



2) Design Poster 3) Choose Focus Group 




6) Redesign Poster 



Figure l.The Posterletgame flow (left) and the selection of positive or negative feedback (right) 


Unlike assessments of knowledge, where the correct answer for 2+2 is 4 and not 5, choice-based assessments 
face a special challenge. To validate choices, we need to establish whether some choices (e.g., seeking 
negative feedback or revising) are better for learning than others. Since there is no literature available on this 
topic, the major goal of the current research is to show that some choices are better than others for 
learning. We will employ different forms of evidence to examine two aspects of validity: 1) Internal. Do 
choices to seek negative feedback and revise correlate with m-gawelearningoutcomes? We employed several 
subsets of learning measures across three different studies. 2) External. Do students who seek negative 
feedback and revise exhibit better in-schooRe anting? We employed academic achievement scores in Study 3. 

Internal Learning Outcomes 

Performancerepresents an in-game category of measures for students’ poster design skills. Coding: We 
employed one measure. Poster Quality , to operationalize performance. The Posterlet game’s graphical 
analysis system evaluates each poster by scoring each of the 21 design principles with 1 if used correctly, 0 if 
not applicable, and -1 if used incorrectly on that poster, producing an individualposter score. Poster 
Qualitysums the individual poster scores of the last poster version on each game level (either the revised 
poster or the initial design, if the poster was not revised) .Poster Quality ranges from -63 to 63. 

Critiquerepresents a posttest category of measures for students’ ability to judge posters. A posttest 
included two open-text questions. The first question asked students to provide some common mistakes a 
poster design novice might make. The second question asked students to provide written feedback on a 
sample poster. Coding: We employed two measures, Common MistakesandWritten Feedback , to 
operationalize critique. We scored each answerby counting the number of graphical design principles 
included in it.Each answer score ( Common Mistakes and Written Feedback) ranges from 0 to 21. 

Principle Selectionrepresents a posttest category of measures for students’ understanding of graphical 
design principles. A posttest included two multiple-choice questions that asked students to choose, from a 
checklist of graphical design principles, the things that were good and bad, respectively, about a sample 
poster. Coding: We employed two measures. Good FeaturesandBad Features, to operationalize principle 
selection. We scored answers by assigning 1 point for each correctly checked answer and subtracting 1 point 
for each incorrectly checked answer. Each answer score (Good FeaturesandBad Features) ranges from -5 to 
5, because each question has five correct and five incorrect answers. 

Recognitionrepresents a posttest category of measures for students’ ability to recognize at a 
glancemisused graphical design principles. A posttest included four sets of questions. For each set, students 
were shown successivelythree images: a poster, then a distractor Moire pattern image, and then a second 
poster, each imagebeing displayed for 5 seconds. Students had to first decide whether the second poster was 
the same/better/worse compared to the first poster and then they had to provide a brief written explanation for 
their decision. We inserted a distractor image between the two posters, since humans display an exceptional 
memory for images (Standing, 1973). This posttest targets the following design rules: the text should not be 
on the poster’s edge, the poster should have images, the images should be relevant to the poster’s theme, and 
the text-background color contrast should be high. The posttest evaluates students on two dimensions: 1) 
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their ability to judge posters (i.e.,to decide whether the second poster was the same/better/worse compared to 
the first poster )and 2)their ability to justify their poster appraisal decision. Coding: We employed two 
measures, Poster RankingandJustify Ranking , to operationalize recognition. Poster Ranking measures 
students’ ability to judge posters .Justify Rankin gmeasmes students’ ability to justify their decisions using 
graphical design principles, differentiating between novice (control) and expert( treatment) students in their 
abilities to perceive subtleties of design principles. Thus, students not only decide whether the posters are 
different, but they can also justify their decision using a new graphical design principle language they learned 
from the game’s feedback. We scoredtheanswer that comparedthe first and the second poster (“same, better, 
or worse”) in each of the four setsof questions by assigning lfor a correct and -1 for an incorrect answer 
(measured by Text not on Edge, Image Present, Graphics Relevant, and Contrast 
High, respectively) . However, we scored Poster Ranking as the count of the correctanswers for each of the four 
poster comparisons (0-4), to be on the same scale as the measure for the justification of the answer. We scored 
eachjustification with 1, if the correct graphical design principle targeted by the question was included in the 
response, and with 0, otherwise. Justify Rankingis the sum of the scores across all poster comparisons (0-4). 

External Learning Outcomes 

Academic achievementrepresents an in-school category of measures for middle-school students’English 
Language Arts ( ELA ), Mathematics (Math), and Scienc e(Science) standardized test achievement. 


5. EXPERIMENTAL OVERVIEW 

We describe three studies using Posterlet to gather evidence on the validity of choice as a learning 
con struct. Study 1 is correlational: itinvestigates whether the choices to seek negative feedback and to revise 
correlate with internallearning outcomes for college students. However, being correlational, it raises the 
concern that students who chose negative feedback could have learned anyway, being successful learners in 
general. Thus, Study 2 is correlational and experimental: it investigates 1) whether Study 1 findings 
replicateeven when the learning measures change and 2) whether the concern raised in Study 1 is 
legitimate.lt compares college students who did not play (Non-players or Control condition) and who 
played( Players or Treatment condition) Posterlet to gauge whether playing the game helps students learn 
design principles. Finally, Study 3 is correlational: 1) it investigates if our findings generalize to other ages by 
sampling middle-school students and 2) it examines external validity through correlations between choices 
and external learning outcomes (in-schoolstandardized achievement). 

5.1 Study 1 

5.1.1 Design 

Participants are N=109 community college students (63 females, 45 males, and 1 not reported) from 
California, aged 15-52, M age =22.13 (SD=5.40). They played Posterlet individually (M=13 min), designing 
three posters, and then took an individual online posttest (M=3 min). We measured: 

Learning Choices .Negative Feedback counts the number of times (out of 9)students choose the “I don’t 
like” feedback, whil eRevision counts the number of revisions (out of 3) across posters. 

Internal Learning Outcomes . /Vv/er Quality measures in-game poster performance, described in Section 4. 
Posttest measures learning of the graphical design principles, adding the normalized scores of the four 
posttest questions in theCritique and Principle Selectioncategories, described in Section 4. 

5.1.2 Results 

Do learning choices correlate with internal learning outcomes? Table 1 shows the correlations among the 
learning choices and internallearning outcomes (poster performance measured by the game and learning of 
the graphical design principles measured by the posttest). Performance ( Poster Quality ) improved across 
levels: Level 1=9.76, Level 2=11.51, Level 3=12.10; F(2,107)=13.46, p<.00 I , so it can also be considered as 
a learning measure.Both choices (Negative Feedback, Revision) correlate with both internal learning 
outcomes (Poster Quality, Posttest), and strongly with each other. Also, Poster gwa/ify correlated strongly 
with the posttest on graphical design principles, providing convergent validity of our learning measures. 
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Table 1. Correlations between negative feedback, revision, and intemallearning outcomes (***p<.001, **p<.01, p<. 05) 


Measures 

Negative Feedback 

RevisionN=109 

Poster QualityN=109 

Posttest N=103 

Negative Feedback 

- 

.52™ 

.21’ 

.20’ 

Revision 


- 

.37*" 

.33** 

Poster Quality 



- 

.44*** 


Negative Feedbackconslatss strongly withRevision.To investigate whether they are unique learning 
predictors, we conducted stepwise linear regressionanalyses using Poster Quality and Posttestas the separate, 
dependent variables. For Poster Quality, onlyRevisionenters as asignificant predictor! P=- 37, F( 1,1 07)= 17. 23, 
p<. 001), accounting for 13.9% of the variance in performance (Adjusted R : =. I Ij.ForPosItesl, Revision is the 
significant predictor! P=- 33, F( 1,101 )=12. 57, /?=.()() I ), accounting for 11.1% of the variance in 
learning! Adjusted R 2 =.10).We also explored the relation between learning choices and internal learning 
outcomes through a series of partial correlations, which revealed that Revision explained bothlearning 
outcomesmore than Negative Feedback did. Thus,, Negative Feedbackand Revz.s/wzpredictall internallearning 
outcomes, and Revision accounts for a little more of the variance in learning outcomes. 

5.1.3 Discussion 

Students seekingnegative feedback performed better on bothinternal learning outcomes and revised more. 
Students who chose to revise also performed better on bothinternal learning outcomes. Flere, 
revisingseemsmore important for learning than seekingnegative feedback, being a strong predictor of 
bothinternal learning outcomes. Finally, students who performed better on the posters also performed better 
on the posttest. 

5.2 Study 2 

5.2.1 Design 

Participants were N=31 students (22 females, 9 males), aged 18-24, M age =20.06 (SD=1.70), from the same 
college as Study 1. They were randomly assigned to one of two conditions, control and treatment. In the 
control condition, N=15 students (11 females, 4 males) took an online individual posttest (M=14 min). In the 
treatment condition, N=16 students (11 females, 5 males) played the Posterlet game individually (M=9 min, 
players were limited to 5 minutes per poster (designing two posters, before taking the same individual online 
posttest (M=13 min). Here, the posttest consisted of the two alternative learning measures, Poster Ranking 
and Justify Ranking, dimensions of theRecognitioncategorydescribed in Section 4. 

5.2.2 Results 

2a. Do learning choices correlate with internal learning outcomes? Table 2 shows the correlations among the 
learning choices and internal learning outcomes for Posterlet players. Performance ( Poster Quality ) improved 
across game levels: Level 1=10.00, Level 2=13.12; Wilks’ Lambda=.66, F(l,15)=7.87, p=. 01. Thus, 
performance can also be considered as a learningmeasure.AiegaO've Feedbackstrongly correlates with 
Revisionand with internal learning outcomes ( Poster Quality , Poster Ranking, and Justify Ranking). Also, 
Poster Quality correlates with both posttest outcomes ( Poster Ranking andJustify Ranking), providing 
convergent validity for thelearning measures. 

Table 2. Correlations between choices and internal learning outcomes for Posterlet players ( p<.01, p<. 05) 


Measures Negative 

Feedback 

Revision 

N=16 

Poster Quality 
N=16 

Poster Ranking 
N=16 

Justify Ranking 
N=16 

Negative Feedback - 

.64"" 

.59’ 

.56* 

.57’ 

Revision 

- 

.37 

.36 

.31 

Poster Quality 


- 

.64** 

.64 


As before, Negative Feedback and Revision are strongly correlated. To investigate their uniqueness as 
predictors of learning, we conducted stepwise linear regressions using Poster Quality and Poster Ranking as 
separate, dependent variables. For both measures, only Negative Feedbacks nters as asignificant predictor: 
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P=.59,F(1,14)=7.61, p=. 01, accounting for 35.2% of the variance inperformance (Adjusted R =.3 1 ) and, 
similarly, P=. 56, F( 1 , 1 4)=6.43, p=.02, accounting for 31.5% of the variance in the Poster Ranking ( Adjusted 
R"=.27).We also explored the relation between learning choices and internal learning outcome sthrough a 
series of partial correlations, which revealed that Negative Feedback explained hothPoster Qualityand Poster 
Ranking more than Revision did. We replicated these results for the alternative posttest measure. Justify 
Ranking. Thus, Negative FeedbackpvccUcisaU internallearning outcomes. Revision, which strongly correlates 
with Negative Feedback, does not reach significance with the small sample size. 

2b. Do treatment students outperform control students on the internal learning outcomes?W e wanted to 
assess whether playing Posterlet improved students’ perception of graphical design principles. A /-test 
comparedposttest learning between students in the treatment and control condition. On Poster Ranking, 
control students (M=1.20, SD=.68) were outperformed [f(29)=-3.74, /?<.() I ] by treatment students (M=2.38, 
SD=1.02).On Justify Ranking, control students (M=1.27, SD=.80)were outperformed [f(29)=-3.52, /?<.() I ] by 
treatment students (M=2.44, SD=1.03).On each individual posttest question, treatment students outperformed 
control students (left side of Figure 2). Figure 2 (left) shows that, on all questions (except for Text not on 
Edge), treatment students performed above chance (i.e., greater than -0.33, because only one out of three 
options is correct, the scores being 1 for correct and -1 for incorrect responses). 

5.2.3 Discussion 

We found the same correlation pattern between seeking negative feedback and internal learning outcomesas 
inStudy 1, although the posttest was replaced. Students who choose negative feedback also tend to revise. 
Although learning choices correlate strongly with each other, only seeking negative feedbackpredicts all 
internal learning outcomes.As before, performance correlates with posttest learning, providing convergent 
validityfor the learning measures.We show that students do learn from Posterlet: they did not already know 
design principles, because treatment students outperformedcontrol students on both dimensions of 
Recognition: they learned to better judge posters andshowed increased perception of the graphical design 
principles. We not only helped students learn, butquite possiblyimproved their perception of the world. 


Study 2: Mean Posttest Question Scores by Condition 

0 Non-players (Control) ■ Players (Treatment) 
j Error bars: +/- 1 SE 



Text not on Edge Image Present Graphics Relevant Contrast High 
Posttest Questions 


Study 3: Mean Posttest Question Scores 

Played Posterlet, then took the posttest 
1 Error bars: ♦/- 1 SE 





Text not on Edge Image Present Graphics Relevant Contrast High 
Posttest Questions 


Figure 2. Study 2 posttest scores by condition (left) and Study 3 posttest scores (right) 


5.3 Study 3 

5.3.1 Design 

Participants were N=976 th -grade middle-school students from California, aged 11-12 years (46females, 
51 males). Students played Posterlet individually, in which they designed three posters, before taking an 
individual online posttest. Only N=80students provided consent and completed Posterlet (M=14 min)and, of 
these, only N=56 students completed the posttest (M=7 min), due to time constraints. We used the same 
choice and performance measures. Here, Posttest combines the measures of the Principle Selection category 
from Study 1 and Poster Ranking from Study 2, described in Section 4, summing their normalized Z-scores. 
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5.3.2 Results 

3a. Do learning choices correlate with internal learning outcomes? N egative Feedback correlates strongly 
with Revision (Table 3). Performance (Poster Quality) improved significantly from the first poster: Level 
1=9.76, Level 2=11.40, Level 3=11.35; Wilks’ Lambda=.87, F(2,78)=6.01, p=.004. Thus, performance can 
also be considered as a learning measure. Only Revision correlates with the internal learning outcomes. 


Table 3. Correlations between negative feedback, revision, and internal learning outcomes ( /x.OOI. **p<.01, “p<. 05) 


Measures 

Negative Feedback 

RevisionN=80 

Poster QualityN=80 

Posttest N=56 

Negative Feedback 

- 

.54*” 

.13 

.22 

Revision 


- 

.23* 

35** 

Poster Quality 



- 

.02 


To investigate the uniqueness of choices as predictors of learning outcomes, we conducted stepwise linear 
regression analyses using Poster Quality and Posttest as separate, dependent variables. Only Revision enters 
as a predictor in the regression:(3=.23, F( 1,78 )=4. 1 9, p=. 04, accounting for 5.1% of the variance in Poster 
Quality (Adjusted R 2 =.04) and. similarly, [>=.35, F( 1,54)=7.65, £>=.008, accounting for 12.4% of the variance 
in Posttest (Adjusted R 2 =.ll).As in Study 1, Revision predicts all internal learning outcomes. Figure 2 (right 
side) shows that, on all questions (except for Text not on Edge), students performed above chance(-0.33). 

3b. Do learning choices correlate with external learning outcomes? Table 4 includes the correlations 
between internal and external learning outcomes. We received achievement scores for many of the students 
to answer this question (samples vary depending on the achievement scores available). Learning choices 
show stable correlations with ELA scores. Additionally, seeking negative feedback correlates with Science, 
while revising correlates with Math. Thus, learning choices in the game(internal)correlate with learning 
outcomes in the school (external). Also, learning on the post test correlates with ELA and Science scores. 
Thus, learning choices measured by the game and learning of the graphical design principles measured by a 
separate post test correlate with external learning outcomes measured by standardized assessments. 

Table 4. Correlations between negative feedback, revision, posttest, and outside assessments (**p<.01, p<. 05) 


Measures 

ELA N=7 1 

Science N=69 

Math N=42 

Negative Feedback 

Revision 

Posttest 

.39** 

.30* 

.42**(N=48) 

.26* 

.18 

.41 **(N=46) 

.30 

.35* 

.30(N=27) 


5.3.3 Discussion 

Students performed above chance on all posttest questions but the first, thus they learned design principles 
playing Posterlet. The first posttest question was challenging for most students, because it targeted a subtle 
design rule that not all students encountered as feedback. Learning choices strongly correlate with each other, 
but only revision predicts internal learning outcomes. More tuning of the posttest is necessary to ensure that it 
measures learning appropriately. Both learning choices in Posterlet predict in-school 
I e a m i n g (/f L4 ) . A d d i t i o n a 1 1 y , seeking negative feedback predicts Science and revising predicts Math. Notably, 
posttest learning ( Posttest ) predicts in-school learningfELA and Science). 


6. CONCLUDING DISCUSSION 

We hypothesized that the choices to seek negative feedback and to revise would predict learning outcomes. 
Across the three studies, we found that the learning choices in the environment correlated with each other and 
with both learning outcomes, correlations between choices and learning outcomes being in the same range (.2 
to .4) for bigger samples. A limitation of the work so far is that, while we find that choices correlate with 
learning outcomes, these relations bounce from Study 1 to Study 3, different choices being better for different 
outcomes. In Study 1 (college students) and Study 3 (middle-school students), revision predicts internal 
learning outcomes (poster performance and posttest learning), while in Study 2 (college students (negative 
feedback predicts internal learning outcomes. For college students (Studies 1 and 2), seeking negative 
feedback correlated with the internal learning outcomes (poster performance and posttest learning), which 
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also correlated with each other, supporting the convergent validity for the learning measures. This result did 
not generalize to middle-school students, likely due to developmental differences (Peters et al., 2014) or to 
the different posttest used in Study 3. Instead, for middle-school children (Study 3), the choice to revise 
correlated with children’s internal learning outcomes (poster performance and posttest learning), as well as 
with their external learning outcomes (standardized achievement scores). For children, in-game choices and 
posttest learning correlated with external outcomes, indicating that choices can predict out-of-game learning. 

We showed through both internal and external validation that choices such as these are beneficial in 
predicting learning, although more research is needed to clarify the contribution of each learning choice to 
learning outcomes. Students who played Posterlet learned to judge posters and to perceive design principles 
better than control students. Thus, we not only helped students learn, but quite possibly improved their 
perception of the world in the context of graphical design principles and poster design. Given these results, 
we showed that choices are important, but that the reliability of the assessment is not yet established. Our 
assessment was focused more on validating the choices rather than the learning outcomes. In summary, we 
developed a choice-based assessment game, Posterlet to track behaviors that we hypothesized are important 
to foster independent learning. We provided a first-of-kind demonstration that choices to seek negative 
feedback and to revise predict better learning in our assessment and in school. Our work may help design 
game environments focused on choices to enhance learning, by embedding learning choices into instruction 
models in both formal and informal environments. 
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ABSTRACT 

Education is a major catalyst in the development of the country. School Based Assessment (SBA) is a new transformation 
in Malaysian education that required subject teachers to conduct formative assessment during teaching and learning 
process according to the procedures by Malaysian Examination Syndicate (MES). Thus, teachers play an important role 
in the implementation of the national education policy in order to develop students' potential and achievement in 
physical, emotional, spiritual and intellectual. This requires teachers’ contributions of effort, involvement, and overall 
professionalization. This study aims to investigate teachers’ readiness, understanding, and workload in implementing 
SBA. Further, this study also seeks to determine the relationship between teachers’ understanding and workload, as well 
as between teachers’ readiness and workload. Participants were comprised of 260 teachers from primary schools in the 
district of Kerian. The results showed that the level of teachers’ understanding and readiness towards implementing SBA 
is high. However, the workload level among teachers was also high. Correlation analysis indicated that there is a 
significant negative relationship between teachers’ understanding and readiness with the level of workload. 

KEYWORDS 

School Based Assessment (SBA), readiness, understanding, workload 


1. INTRODUCTION 

The success of an education system is determined by students’ learning and performance. Ministry of 
Education (MOE) in Malaysia realized that the education system needs to go through a comprehensive and 
systematic transformation if Malaysia aspires to produce quality individuals who are able to compete in a 
global market (Preliminary Report of Malaysian Education Development Plan 2013-2025, September 2012). 
In line with the National Education Philosophy, MOE focuses on the holistic development of students that 
emphasizes the development of intellectual, spiritual, emotional, and physical. 

Malaysia has undergone few educational policies in order to improve students’ development and 
performance. The country has implemented New Primary School Curriculum in 1983. However, the policy 
name has changed to Primary School Integrated Curriculum (KBSR) in 1993 (KPM, 2014). The assessment 
in KBSR was more on examination oriented instead of holistic education (Lembaga Peperiksaan Malaysia, 
2012). Therefore, the policy has been reviewed and new educational policy. Primary School Standards 
Curriculum (KSSR) was introduced in 2012. KSSR emphasized on holistic education including reading, 
writing, counting, reasoning, ICT, development of socio -emotion, spiritual, physical, cognitive, behavior and 
value (KPM, 2014). In line with KSSR, government announced National Education Assessment System 
(SPPK) and under this system. School Based Assessment (SBA) was introduced (JPN Perak, 2013). The 
main objectives of SPPK are to reduce focus on examination, strengthen SBA, improve students’ learning, 
continuous holistic assessment, and develop better human capital (Lembaga Peperiksaan Malaysia KPM, 
2012 ). 

There are five components of SPPK including the assessment of physical activities (sports and co- 
curriculum activities), psychometric assessment, school assessment, centre assessment, and centre 
examination. School assessment is divided into two parts: formative and summative (Lembaga Peperiksaan 
Malaysia KPM, 2012). For school assessment, each school is responsible for implementing their own 
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assessment, which requires teachers to design, construct items and instruments, manage, mark the scores, 
record and report the assessment for every subject they teach. The aims of school assessment are to enhance 
students learning and improve teaching effectiveness (Lembaga Peperiksaan Malaysia KPM, 2012). 
Furthermore, a few characteristics of the SBA including: (a) able to provide a holistic overview of the 
knowledge and skills attained by students, (b) continuous assessment of teaching and learning, (c) flexible 
assessment methods according to students’ ability and readiness, and (d) view students’ achievement based 
on performance standards. The performance standard is a set of statements describing the achievement and 
mastery of an individual student within a certain discipline, in a specific period of study based on an 
identified benchmark. The performance standard will help inform teachers the most suitable way to assess 
individual student fairly in a focused manner based on the predetermined set of standards. 

The evolution of national education system requires a paradigm shift among teachers. Drastic changes 
need to be made by the teachers to adopt the new education system. They must change the way they think 
and practice (Zaidatun & Lim, 2010). In every education plan, teachers play very important roles and they 
must fully understand on the implementation of the new system (Nor Hasnida, Baharim & Afian, 2012; 
Sanitah & Norsiwati, 2012). 


2. PROBLEM STATEMENT 

The education sector in Malaysia demands high commitment among teachers to plan lessons, teach in the 
classroom, prepare students’ report card, conduct co-curriculum activities, attending professional 
development courses, and collaborate with parents and the community. This is supported by Lemaire (2009) 
who found that teachers are burdened with tasks that unrelated to teaching and learning, extra-curricular 
activities, attending meetings, conducting student programs, and managerial duties. Furthermore, various 
reforms of education system contribute to teachers’ stress as they are facing challenges and pressures to 
fulfill the requirements of the new system (Tajulashikin, Fazura & Mohd Burhan, 2013). 

In 201 1, a survey by the MOE found that teachers work 40 to 80 hours per week, with an average of 57 
hours (Preliminary Report of Malaysian Education Development Plan 2013-2025, September 2012). In 
addition. National Union of the Teaching Profession (NUTP) has complained about teachers’ workload to the 
Education Minister on 30 March 2010 (Berita Harian, 1 April 2010). As SBA announced in 2012, teachers’ 
workload increases as they have to conduct the assessment process from beginning. They have to key in the 
marks online in the School Based Assessment Management System (SPPBS). This means that teachers’ 
understanding and commitment is crucial as they are empowered to assess their students (Md Noor & Sahip, 
2010). Moreover, the large numbers of students in one class contributes to the difficulty in assessing every 
student (Harakah, 15 July 2013; Berita Harian, 15 Disember 2010). 

According to Maizura (2010), readiness is an important aspect in determining the success and failure in 
implementing changes in the curriculum. For instance, Malaysian has implemented PPSMI policy in 2003. 
The policy required teachers to conduct Sciences and Mathematics in English language. This has raised 
objections from various parties. Finally, PPSMI policy was announced to be discontinued in 2012 and 
replaced by the policy of upholding the Malay language and strengthening the English language (MBMMBI). 
The implementation of PPSMI was considered unsuccessful because most of the teachers were not fully 
equipped with English language skills (Nor Safiza, 2011). Students also not ready to learn science and 
mathematics in English. Therefore, the purpose of this study is to investigate the level of teachers’ 
understanding, readiness and workload in implementing the SBA among primary school teachers in one 
district. The present study also seeks to determine the relationship between understanding, readiness and 
workload. 
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3. METHODOLOGY 

This study involved 260 teachers from 67 primary schools in Kerian district. A stratified sampling technique 
was used, where the researcher divided the schools into National Primary School (SK) and National-type 
School (Chinese and Tamil). Participants were randomly selected from each school. The instrument used in 
this study was adapted from previous research conducted by Kalawathi (2013), Nesan (2012), Fazura (2011) 
and NUTP Survey. The questionnaire is divided into three parts: demographic factors, teachers’ 
understanding and readiness, and teachers’ workload in implementing SBA. The questions are 5-point Likert 
scales ranging from disagree very much to agree very much. The reliability of the instrument has been 
verified by Cronbach’s Alpha, in which alpha value for understanding is 0.893, readiness is 0.831, and 
workload is 0.792. 

The data gathered from the participants was analysed using the Statistical Package for the Social Sciences 
(SPSS). Descriptive statistics were used to discuss the participants’ demographic information and 
participants’ level of understanding, readiness, and workload. The differences between participants’ level of 
workload and gender and school category was analysed using Independent-samples t test. One-way ANOVA 
was utilized to determine the differences between participants’ level of workload and school type. The 
relationship between participants’ understanding and workload, as well as the relationship between 
participants’ readiness and workload were measured by correlation analysis. 


4. RESULT 

This section will discuss about the demographic factors of the participants, the level of understanding, 
readiness and workload in implementing SBA. The result of the differences between level of workload and 
gender, school category and school type will be discussed too. Further, this section will reveal the findings on 
the relationship between level of understanding and workload, and relationship between level of readiness 
and workload among teachers in implementing SBA at school. 


Table 1 . Demographic factors of the participants. 




Frequency 

Percentage (%) 

Gender 

Male 

75 

28.8 


Female 

185 

71.2 


Total 

260 

100 

School category 

Rural area 

159 

61.2 


Urban area 

101 

38.8 


Total 

260 

100 

School type 

National (SK) 

164 

63.1 


National type-Chinese (SJKC) 

40 

21.5 


National type- Tamil (SJKT) 

56 

15.4 


Total 

260 

100 


As refer to Table 1, majority of the participants are female (71.2%). Male teachers are only 28.8% of the 
the total participants. According to school category, most of the participants are working at schools in rural 
area (61.2%). Those working at the urban area is only 38.8%. There are 63.1% of the participants working at 
SK, 21.5% at SJKC, and 15.4% at SJKT. 
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Table 2. Level of understanding, readiness, and workload 



Level 

Mean 

Frequency 

Percentage (%) 

Understanding 

Low 

1.00-2.33 

14 

5.4 


Moderate 

2.34-3.66 

62 

23.8 


High 

3.67-5.00 

184 

70.8 


Total 


260 

100 

Readiness 

Low 

1.00-2.33 

6 

2.3 


Moderate 

2.34-3.66 

69 

26.5 


High 

3.67-5.00 

185 

71.2 


Total 


260 

100 

Workload 

Low 

1.00-2.33 

0 

0 


Moderate 

2.34-3.66 

71 

27.3 


High 

3.67-5.00 

189 

72.7 


Total 


260 

100 


Table 2 describes the level of understanding, readiness, and workload in implementing SB A among 
primary school teachers. The majority of the participants has a high understanding about the SBA (70.8%) 
and only 5.4% of them have low understanding. For readiness aspect, most of the participants have a high 
level of readiness (71.2%). However, there are 26.5% of them have a moderate level of readiness. Most of 
the participants respond that they have a high level of workload (72.7%). This is followed by 27.3% of them 
that reported moderate levels of workload. However, none of the participant rated low level of workload. 

One way ANOVA found that there is a significant difference between the level of workload and school 
type, F (2, p=0.000) = 40.260. Post-hoc test indicated that significant differences exist between SJKC, SJKT 
and SK. Participants at SJKC reported the highest level of workload (M=4. 3708, SD=0. 3954). 

As displays in Table 3 below, there is a significant difference between the participants’ level of workload 
and gender, t (258, p=0. 000) = -4.032, p<0.05. Both male and female teachers reported high level of 
workload. However, female teachers reported higher workload level (M=3.92, SD=.4344) than their male 
counterpart (M=3.68, SD=.4703). The result also shows that there is a significant difference between the 
participants’ level of workload and school category, at (258, p=0.024) = -2.278, p<0.05. Although teachers in 
rural and urban area reported high level of workload, teachers in rural area rated higher level of workload 
(M=3.90, SD=.4479) than those in urban area (M=3.77, SD=.4640). 


Table 3. Differences between participants’ level of workload and gender and school category 



N 

Mean 

SD 

t value 

Df 

P 

Gender 







Male 

75 

3.68 

.4703 




Female 

185 

3.92 

.4344 

-4.032 

258 

**.000 

School Category 







Rural area 

159 

3.90 

.4479 




Urban area 

101 

3.77 

.4640 

-2.278 

258 

**.024 


** Significant at 0.05 


Pearson correlation analysis revealed that there is a significant negative relationship between the 
participants’ level of understanding about the SBA and their workload (r = -.216, p < 0.01). The result also 
found that there is a significant negative relationship between the participants’ level of readiness for 
implementing SBA and their workload U-.266, p<0.01). These indicate that the higher perceived level of 
participants’ understanding and readiness about the SBA, the lower they would perceive of their workload. 
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5. DISCUSSION 

Overall results indicated that the majority of the teachers at the particular district understand about the 
implementation of the SBA. They were also ready in implementing the system. However, as the SBA is a 
new assessment system introduced by the Malaysian government, majority of the teachers reported high level 
of workload in implementing it. Teachers at SJKC perceived highest level of workload as compared to 
teachers in other school types. In addidition, teachers in rural area experienced higher level of workload than 
those in urban area. This may due to the lack of the facilities and internet coverage at the rural area. In terms 
of gender, female teachers percieved higher level of workload in implementing the SBA as compared to their 
male counterpart. 

Findings showed that teachers in primary schools have a high level of understanding in implementing the 
SBA. Items analysis indicates that most teachers are benefited from attending training sessions organized by 
MOE in which it managed to increase their understanding about SBA. In addition, support services and 
online mentoring by the Malaysian Examination Syndicate is also helpful in improving their understanding 
about SBA. Besides, their readiness level is high in implementing SBA at school. Other research conducted 
by Ruhila (2012) and Ismadiah (2012) also found that the level of understanding is high among teachers in 
Johor. However, teachers’ readiness in implementing SBA is only at a moderate level (Ismadiah, 2012; Nor 
Hasnida et al., 2013). Items analysis shows that teachers make a good preparation before teaching the KSSR 
subjects and always implement innovations in teaching and learning. However, teachers are not highly ready 
to try new strategies in implementing SBA. Furthermore, their level of readiness to go for more training 
about SBA was moderate. 

This study found that the level of teachers’ workload is high in implementing SBA. Factors that 
contribute to their high level of workload are SPPBS is difficult to access, delay databases due to poor 
internet coverage in rural schools, and they need to print a lot of instruments for evaluation purposes. In 
addition, the evaluation of ICT elements is difficult due to lack of computer facilities in schools. 
Furthermore, teachers face difficulties in evaluating weak students and those who always absent from school. 

The results indicated that there are negative relationship between the levels of understanding and 
workload, and between the level of readiness and workload. Teachers who have low understanding and 
readiness about the SBA tend to have higher level of workload. According to Mahamod, Yusoff and Ibrahim 
(2009), teachers are the driving force and the main impetus to the process of teaching and learning in the 
classroom. Therefore, a teacher must be equipped with all related knowledge of the SBA. Implementation of 
formative assessment in the SBA requires serious changes among teachers. They need to change the 
perception of their role in improving student achievement and classroom practice (Nesan, 2012; Hamzah & 
Sinnasamy, 2009). 

SBA as a new reform in the Malaysian education system requires educational leaders to take proper 
approaches in managing the changes. Kurt Lewin proposed that successful change should follow three steps: 
unfreezing the status quo, movement to the desired end state, and refreezing the new change to make it 
permanent (Robbins & Judge, 2013). According to Osland, Kolb, Rubin, and Turner (2007), resistance to 
change is a natural reaction to change and part of the process of adaptation (p. 643). Hence, unfreezing the 
status quo refers to overcome the pressures of both individual resistance and group conformity (Robbins & 
Judge, 2013, p.619). This could make the transformation process from status quo to desired aims. The 
restraining forces that hinder movement from the existing equilibrium should be decreased. Management 
should focus on how to increase driving forces that direct behavior away from the status quo (Robins & 
Judge, 2013). Thus, refreezing step stabilizes a change intervention by balancing driving and restraining 
forces (Robbins & Judge, 2013). In implementing the SBA in schools, the educational leaders should 
understand factors that affecting teachers’ understanding, readiness and workload. Highly understanding and 
readiness among teachers will decrease their perceive workload in implementing the new system. 


6. CONCLUSION 

The SBA was introduced in 201 1 to be implemented in all primary schools in Malaysia. Over the years, it is 
still early to evaluate the progress and success of the new system. The attitude of teachers who are not 
comfortable with the reform needs to be addressed. Exam-oriented emphasis needs to be changed to the 
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assessment of individual student’s skills and achievement. Implementation of formative assessment in SBA 
requires a serious shift in mindset among teachers. Furthermore, teachers will be burdened and distressed in 
implementing the SBA if they do not understand about the system and if they are not ready to fulfill it. 
Teachers need to be more skilled in time management. Among the challenges in implementing the SBA is the 
use of on-line reporting system, management of document files, assessment of students who have different 
competencies in the classroom, teaching and learning strategies, and allocation of time for the 
implementation of activities in the classroom. Therefore, support from principals is crucial in implementing 
the SBA in schools. According to Muzammil and Kamariah (2011), good interaction between the school 
principals and teachers will contribute to higher level of job satisfaction and better performance among 
teachers. In addition, the management of the school should provide great assistance and adequate facilities 
for teachers in order to implement this new system. 

As a conclusion, teachers in primary schools in the district of Kerian have a high level of understanding 
and readiness in fulfilling the SBA. However, they are burdened with high workload in implementing it. 
Since the significant negative relationship exists between level of understanding, readiness and workload, 
possible way to reduce the workload is to ensure that teachers highly understand the requirements of the SBA 
and are always ready to implement it. Consequently, they will also change their perception to a more positive 
view of the SBA. 

Findings from this study would be able to provide important information to MOE, schools and teachers 
concerning the implementation of SBA in schools. Revealed aspects such as the level of teachers’ 
understanding, readiness, workload, and the relationship between understanding and readiness to workload, 
provide better insights on how to effectively and efficiently implement this new assessment system. 
However, as the findings of this study cannot be generalized, more study is needed in order to contribute to 
the existing knowledge relating to SBA implementation in Malaysia. As the Malaysian government 
continuously makes an improvement of this new assessment, research on the level of parents’ acceptance and 
the effectiveness of the system in students’ performance should be conducted. 
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ABSTRACT 

The purpose of this study is to use the Analytic Hierarchy Process in order to identify how frequency of media use in 
daily life affects decision-making in media choice. 276 university students took part in this research. They were asked to 
prioritize their ways of obtaining information about current affairs using sets of media such as TV, books, newspapers, 
webpages, Twitter and Facebook. Multiple regression analysis showed a clear relationship between frequency of media 
use in daily life and decision- making in media choice. It was suggested that frequency of media use in daily life affects 
decision- making in media choice. For example, even in aspects where it would be appropriate to obtain information from 
books, the high web using group tends to select the web. Information about these students’ tendencies is useful as meta- 
cognitive knowledge. 
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I. INTRODUCTION 

In recent years, the development of Media Literacy, i.e. the ability to make judgments about the reliability, 
bias and accuracy of information, has been considered to be a crucial factor in media education (DCMS2001, 
Buckingham 2003, Buckingham 1998). The OECD (2009) emphasizes the importance of bringing critical 
reading skills to bear on electronic texts. Students use various types of media in the classroom, at home and 
in their own rooms, such as TV, books, newspapers and the Internet(Hashimoto,201 l;Hashimoto et al 2011). 
Each type of media has its own particular characteristics, and an understanding of media characteristics is an 
important factor in media literacy. For example, let us consider the massive earthquake in Japan on March 

II, 2011. Many different types of media provided a wide variety of information about the danger of the 
tsunami and the situation in the nuclear power station at Fukushima. Some people believed false information 
provided by Internet. 

We were once again made aware of how important it is to gather information based on an understanding 
of the characteristics of different types of media. Sakamoto (1986) pointed out that critical viewing skills and 
an understanding of the characteristics of different media are the major constituents of media literacy. The 
DCMS (2001) emphasized the importance of being able to explain one's reasons for using a particular type of 
media on the basis of one's understanding of the characteristics of that type of media. 

In this study the author focuses on decision-making in media choice. In our daily life we have to select 
various kinds of media, such as television, books, newspapers, magazines and Internet. The author used the 
Analytic Hierarchy Process (AHP) as a tool for simulating students' decision-making in their use of media. 
AHP is a structured technique for organizing and analyzing complex decisions. Using AHP, students have to 
define appropriate criteria and priorities for using media. 

However, it became clear that there is a great deal of difference among individuals when it comes to 
decision-making in media choice(Gotoh,2012a;2012b;2013,2014). It was discovered that some students 
prioritize ease-of-use and do not prioritize accuracy in the context of obtaining information to write a report. 
Some students prioritize the Internet and do not prioritize books in the same context. This trend could even 
be called extreme in the case of certain students. The frequency of media use in daily life seems to be an 
influential factor. Research into media preconceptions showed that experience of media use affects 
preconceptions about media. For example, students at schools that are not connected to the Internet feel that 
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watching television is easy and using the Internet is hard, whereas students at schools that are connected to 
the Internet feel that both watching television and using the Internet are easy. A lack of use of specific media, 
such as little reading experience, seems to affect students’ decision-making in media choice. Studies have 
been done on the relationship between experience of media utilization and preconceptions about media. 
However, the effects of frequency of media use on decision-making in media choice are unknown. 


2. PURPOSE 

The purpose of this study is to use the Analytic Hierarchy Process in order to identify how frequency of 
media use in daily life affects decision-making in media choice. 


3. METHODS 

3.1 Analytic Hierarchy Process 

276 university students (pre service teacher education, physical therapy, nursing, nutrition, social welfare) 
took part in this research. They were asked to prioritize their ways of obtaining information about current 
affairs using sets of media such as TV, books, newspapers, webpages, Twitter and Facebook. Fig.l shows 
AHP hierarchy. 

• Objective: to identify the ideal balance in the use of different media in order to obtain information about 
current affairs for business and study. 

• Criteria: accuracy, timeliness, enjoyment, ease-of-use and search function. 

• Alternatives: TV, books, newspapers, webpages, Magazines. 

Esumi Excel Conjoint Analysis/ AHP Ver. 1 was used as the analysis tool. Students were asked to reply to 
questionnaires and the paired comparison method was used. The visualization of media characteristics can 
display not only average entire samples but also individual students. Students were asked to write reflective 
reports as they viewed their individual visualizations. Overall tendencies and some individual cases are 
examined on the basis of these data. 


Objective 


Criteria 


Alternatives 



Figure 1. AHP hierarchy 
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3.2 Frequency of Media Use in Daily Life 

Items were developed by referring to the NHK Broadcasting Culture Research Institute and to research 
conducted by the University of Tokyo. 

• Reading books 1: never, 2: 1 or 2 books per month, 3: 3 to 5 books per month, 4: 6 books or more 
per month. 

• Using web 1: never, 2: about 10 minutes per day, 3: 11 to 30 minutes per day, 4: 31 to 60 minutes 
per day, 5:1 hour or more per day. 

• Watching TV 1: never, 2: about 30 minutes per day, 3: 31 to 59 minutes per day, 4: 1 hour to 3 hours 
per day, 5: 3 hours or more per day. 

• Reading newspapers 1: never, 2: about 5 minutes per day, 3: 6 to 15 minutes per day, 4: 16 to 30 
minutes per day, 5:30 minutes or more per day. 

• Reading magazines 1: never, 2: 1 or 2 magazines per month, 3: 3 to 5 magazines per month, 4: 6 
magazines or more per month. 

3.3 Analysis 

Spearman's rank correlation coefficient was used in order to analyze the part played by frequency of media 
use for each type of media. To identify how frequency of media use in daily life affects decision-making in 
media choice, the sample in each type of media was divided into two groups, a high-frequency group and a 
low-frequency group. The priority of accuracy, timeliness, enjoyment, ease-of-use and search function for 
each type of media was totaled and used as the score for decision-making in media choice. Multiple 
regression analysis was used to determine the association between frequency of media utilization in daily life 
and decision-making in media choice. 


4. METHODS 

4.1 Frequency of Media Use 

On the basis of the data, the entire sample was divided into two groups for each type of media. Reading 
books: never (62.5%) is taken as the low group, and the rest are taken as the high group for reding books. 
Using web: 31 to 60 minutes per day (15.5%) and 1 hour or more per day (40.1%) are taken as the high group 
and the rest as the low group for using web. Watching TV: 1 hour to 3 hours per day (16.6%) and 3 hours or 
more per day (43%) are taken as the high group and the rest as the low group for watching TV. Reading 
newspapers: Never (80.5%) is taken as the low group and the rest as the high group for reading newspapers. 
Reading magazines: Never (49.1%) is taken as the low group and the rest as the high group for reading 
magazines. 

Spearman's rank correlation coefficient was calculated in order to view the part played by frequency of 
media use for each type of media(See Table 1). Statistically significant positive correlations were shown 
between reading books and using the web, and reading books and reading newspapers. Statistically 
significant negative correlations were shown between reading books and watching TV, using the web and 
watching TV, reading newspapers and reading magazines, and watching TV and reading magazines. The 
rank correlation coefficient among predictor variables is not particularly high, and there is no problem of 
multi-collinearity. 
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Table 1. Correlation coefficient between frequency of media use 



Reading 

books 

Using web 

Watching 

TV 

Reading 

newspapers 

Reading 

magazines 

Reading books 

1 

.164** 

-.208** 

.142* 


Using web 


1 

-.151** 



Watching TV 



1 


.161** 

Reading 

newspapers 

Reading 

magazines 




1 

.115* 

1 


**p<.01, *p<.05 


4.2 Decision-Making in Media Choice 

C.I. (consistency index) of all samples are as follows: criteria (0.003), accuracy) 0.0025) , timeliness(0.025), 
enjoyment (0.0052), ease-of-use (0.0136) and search function(0.0024). Each C.I. is lower than 0.15, hence 
there is no problem of internal consistency. 

Concerning criteria, in all samples, the top priority is accuracy (0.358) , followed by ease-of-use 
(0.185), search function (0.184), timeliness (0.128) and enjoyment (0.144). Concerning alternatives, in all 
samples, the top priority is using the web (0.293), followed by watching TV (0.227), reading newspapers 
(0.199), reading books (0.154) and reading magazines (0.125). These tendencies are similar to the results of 
previous research (Gotoh2012a, 2012b). 


0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 


Reading books 


Using web 


Watching TV 


Reading 

newspapers 


Reading 

magazines 


□ Timeliness □ Accuracy □ Ease-of-use 0 Enjoyment a Search function 


Figure 2. Decision-making in media choice (All samples) 
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The table2 shows the correlation coefficient between criteria. Statistically significant negative 
correlations were shown between accuracy and ease-of-use, accuracy and timeliness, accuracy and 
enjoyment, accuracy and search function, timeliness and ease-of-use, timeliness and enjoyment, ease-of-use 
and enjoyment, and enjoyment and timeliness. These tendencies are also similar to the results of previous 
research (Gotoh2012a, 2012b). 


Table 2. Correlation coefficient between criteria 



Accuracy 

Timeliness 

Ease-of- 

use 

Enjoyment 

Search 

function 

Accuracy 

1 

-.263** 

-.372** 

_ 46i** 

-.473** 

Timeliness 


1 

-.150 ** 

-.187** 


Ease-of-use 



1 

-.165** 


Enjoyment 




1 

-.131* 

Search function 





1 






f p<.01, * p<.05 

4.3 Relationship 

between 

Frequency 

of Media 

Use in Daily 

Life and 


Decision- Making in Media Choice 

The table shows the results of multiple regression analysis and partial regression coefficient (P). Forced entry 
was used. The high group in reading books gives priority to reading books. The high group in using the web 
gives priority to using the web. The high group in watching TV gives priority to watching TV. The high 
group in reading magazines give priority to reading magazines. To summarize, students tend to select 
familiar media even in an information- gathering situation in order to write an academic report. 


Table 3. Results of multiple regression analysis and partial regression coefficient (|3) 



Books 

Web 

TV 

Magazine Newspaper 

Reading books 

.301*** 

-.031 

-.268*** 

-.003 

.055 

Using web 

-.064 

.267*** 

-.11 + 

-.124* 

-.069 

Watching TV 

+ 

oo 

o 

-.074 

.227*** 

-.020 

-.068 

Reading magazines 

-.221*** 

-.057 

.080 

.230*** 

.070 

Reading newspapers 

-.221 

.043 

.061 

-.122* 

.006 

R 2 


.085** 

.197 + 

.075** 

.013 


+p<.10, *p<.05, ** p<.01, *** p<.001 
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5. DISCUSSION 

Multiple regression analysis showed a clear relationship between frequency of media use in daily life and 
decision- making in media choice. It was suggested that frequency of media use in daily life affects decision- 
making in media choice. For example, even in aspects where it would be appropriate to obtain information 
from books, the high web using group tends to select the web. 

Information about these students’ tendencies is useful as meta-cognitive knowledge. Students use various 
types of media in the classroom, at home and in their own rooms in digital era. He or she will choose media 
without awareness. Meta-cognitive knowledge is useful appropriate media choice and reflection. Students 
should be aware of their media use tendencies and select the appropriate type of media without regard to 
frequency of media use (Gotoh.Ikuta & kurokami, 2009). Visualization of daily media utilization is also 
helpful. For instance, the media diary method, where frequency of media use is recorded and visualized, is 
helpful. 

In the near future, the author wishes to develop a method which can provide meta-cognitive knowledge 
about frequency of media use in daily life. 
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EXPLORATION OF DATA IN LINE GRAPHS 
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ABSTRACT 

This research compared undergraduate Novices and PhD Experts in psychology and business in their exploration of 
psychology and business domain graphs. An overall expertise effect in graph explanation was found. Results indicated 
that Novices paused longer than Experts before beginning their explanations. Qualitative analyses showed that Experts 
were generally more complete in their explanations, generating more inferences, more quantitative statements, and more 
conceptual messages. Psychology Experts tended to generate more complete explanations for psychology-domain graphs 
whereas Business Experts generate less complete explanations for business-domain graphs. The results suggest that 
Experts have superior strategies to Novices in graph exploration that may be accommodated by the graph comprehension 
model of Pinker (1990). An implication of these results is that simple instructions may greatly enhance the data literacy 
of students and might be embodied in data visualization tools for adults and researchers as well. 


KEYWORDS 

Graphs, expertise, experts, novices, data 


1. INTRODUCTION 

Graphs are among the most effective ways for people to understand data (Tufte, 1983). Often the purpose is 
straightforward communication of data as might be found in school textbooks or newspapers or internet sites 
(Roth, Bowen, & McGinn, 1999). Exploration is a special and very interesting case of graph use (Behrens, 
1997), often representing a person’s attempt to understand, interpret or communicate data. While a common 
application of graph exploration is in scientific reasoning, it is difficult to imagine many domains where 
graphs intended for exploratory purposes are not found (e.g., Bertin, 1983; Kosslyn, 2006). 

Curiously, unlike many other domains such as chess and physics, (Eriksson, 2005) graph exploration does 
not appear to demonstrate a consistent difference between experts and novices. This is unfortunate because 
confronted with a graph that requires people to utilize complex inferential processes, a number of interesting 
theoretical and practical questions arise: Do experts apply qualitatively different strategies than novices (e.g., 
Gick & Holyoak, 1983)? Do novices focus on the graph’s syntactic structure at the expense of an analysis of 
the deeper semantic components (e.g., Preece & Janvier, 1993)? Are experts able to recognize patterns in 
graphs in ways that may be similar to how expert chess players recognize chess positions (e.g., Newell and 
Simon, 1972)? Can graphical visualization tools be designed to better facilitate novice understanding (e.g. 
Konold, 2007)? More generally, what differences do experts and novices exhibit in graph exploration? 

Relatively few studies have addressed the issue of expertise in graph exploration directly. One instance is 
the ethnographic research of Roth and Bowen (2003) who examined how domain experts in biology, physics, 
and forest sciences interpreted familiar and unfamiliar graphs. Roth and Bowen found that experts had 
significant difficulty interpreting graphs taken from undergraduate textbooks from their respective domains 
but they had little difficulty with familiar graphs taken from their own personal research. 

In a different domain, Trafton et al (2002) described how expert meteorologists create spatial 
transformations of meteorological data when the information requested of them is not explicitly present. For 
example, in determining the air pressure over Pittsburgh, Trafton et al.’s eye movement data suggested that 
participants were identifying nearby isobars, calculating the distance between them, and then using the 
proportional distance to calculate the atmospheric pressure. 
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However, neither the Roth and Bowen nor the Trafton et al. studies directly compared experts against 
novices in their respective domains. Thus, it is difficult to know if the strategies inferred by these authors 
were attributable to expertise per se or were idiosyncratic to the domains selected. 

Freedman and Shah (2002) conducted one of the few studies to explicitly compare domain-specific expert 
(psychology graduate students) and novice (undergraduate students) graph exploration. Freedman and 
Shah’s domain-specific graphs included graphs on cognitive studies of aging whereas domain non-specific 
graphs were concerned with non-cognitive aging data. Freedman and Shah reported that novices tended to 
describe main effects while experts were more likely to describe the underlying mathematical functions in the 
graph stimuli. However, the domain manipulation had no effect. Freedman and Shah interpreted these results 
as supporting the notion that novices attend to lower-level perceptual features of a graph whereas experts 
enrich and elaborate the visual features of a graph with their domain knowledge. However, it is difficult to 
reconcile Freedman and Shah’s results with those of Roth and Bowen (2003). Is expertise in graph 
exploration a general skill (as suggested by Freedman & Shah) or one that is very specific to a given expert’s 
domain (like those of Roth & Bowen)? 

The purpose of the current research is to identify differences (if any) between novices and experts in their 
exploration of graphs drawn from familiar and unfamiliar domains. If experts are superior to novices 
regardless of domain, then graph expertise may be a more general ability. The contribution of this research 
speaks not only to our understanding of expertise but also to the application of data visualization tools and to 
the education of students from different disciplines in terms of their understanding of data. 

Shah and Carpenter (1995) compared psychology graduate- and undergraduate students using graphs 
from common-knowledge domains. They found no effect of expertise. Using business and psychology 
domain-specific graphs and PhD faculty Experts compared to undergraduate Novices, the present study was 
designed as a more sensitive test of graph expertise, leading to Hypothesis 1: Experts would generate more 
causal inferences about graphs than Novices. 

Expertise tends to be domain-specific. However, the role of domain-specificity as a function of expertise 
has not been investigated in graph exploration studies before and this formed Hypothesis 2: Experts would 
provide more complete explanations of graphs in familiar than in unfamiliar domains. 

Carpenter and Shah (1998) found the proportion of nominal, ordinal, and metric descriptions of graphs 
varied across different graph types. Nominal utterances were defined as the names of 7-variables without any 
ordinal or metric information about the z-y relation; ordinal utterances mentioned the explicit relationships 
between z-variables; and metric utterances included descriptions of the interval or ratio relationship between 
7-variables. Equating Carpenter and Shah’s nominal, ordinal, and metric descriptions with the different types 
of conceptual messages proposed by Pinker (1990) we may be able to extend Pinker’s model to include 
expertise and which leads to Hypothesis 3; Experts would generate more conceptual messages (nominal, 
ordinal, and metric combined) than Novices. 

In order to understand how expertise might exert its effects on graph exploration and to better control for 
potential floor- and ceiling effects, both simple and complex graphs were employed. Somewhat more 
complex graphs might allow Experts to demonstrate superiority over Novices, as predicted by Hypothesis 4; 
Experts will provide more complete graph explanations than Novices. 


2. METHOD 
2.1 Participants 

Twenty-six participants were recruited from the Carleton University community. Out of ten (seven female) 
undergraduate Novices, six were majoring in psychology and four in business. The Expert sample comprised 
eight psychology (seven female) and eight business (six female) PhD faculty. Five Novices, six business 
Experts, and seven psychology Experts reported that they had to create graphs and all reported that line 
graphs were the graphs most familiar to them. Novice undergraduate students were granted 1.0% course 
credit, and Experts were given a $10 coffee shop gift certificate for their participation. All had normal or 
corrected-to-normal vision. Participants were tested individually in sessions lasting a mean of 75 minutes. 
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2.2 Apparatus & Materials 

Ten, three -point, two 7-variable line graphs were used, five simple and five more complex as determined 
through pilot testing. Each graph was assigned two sets of titles, labels and axes; one drawn from psychology 
and one from business. The business labels were selected from an undergraduate textbook on international 
business (Griffin & Pustay, 2007), and psychology labels were drawn from an undergraduate textbook on 
psychology (Weiten & McCann, 2007). The 10 business graphs were the mirror images of the 10 psychology 
graphs as shown in the typical examples in Figure 1 below. 



Exports As A Percentage of GDP 2006-2008; 
Brazil versus Germany 


Percentage of GDP 

O 

S 

\ 


2006 2007 2008 


Number of Publications By Year By School of 
Thought 


\ ■ Behavioural 

| \ 
z \ 

\ Psychoanalytic 


2006 2007 2008 


Figure 1 . Example graph stimuli: business domain (top) and psychology domain (bottom); “Describe” (left) and 

“Explain” (right) 

Stimulus presentation was randomized, controlled by DirectRT™ on a Dell Latitude D610 laptop 
computer with 1280 X 800 pixel screen resolution. Participant verbalizations were recorded on a Panasonic 
RR-US500 digital voice recorder. 

2.3 Procedure 

After Preliminary instructions and Informed Consent, detailed experimental instructions were provided. Four 
practice trials were followed by 20 experimental trials, each initiated by pressing the spacebar. On each trial a 
randomly selected graph without labels or titles was displayed with the word “describe” played over the 
computer speakers as well as appearing at the bottom of the display. (Pilot testing had indicated that alerting 
participants to the visual characteristics of a graph was important in order to prime their subsequent 
explanations.) When done, participants were instructed to press the spacebar whereupon the graph was re- 
displayed with the corresponding business or psychology labels and titles accompanied by the instruction 
“explain” played over the computer speakers and displayed on the screen. At the end of the experiment 
participants were debriefed, thanked, and paid (if applicable). 
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2.4 Data Analysis 

Verbal protocols were transcribed ad verbatim, coded, and analyzed with NVIVO™ Version 8.0. Frequency 
of utterance-type was calculated, as was the presence/absence and completeness of explanations and the 
frequency of conceptual messages (sum of nominal, ordinal, and metric utterances). Interrater reliability was 
assessed by an independent rater coding a randomly selected 15% of the verbal protocols and percentage 
agreement was 90.0%. 


3. RESULTS 

Coding of the graph explanation protocols resulted in nine themes, shown for each expertise group in Table 
1. Values are proportions of the total number of trials per expertise group to enable direct comparison of the 
different groups. Because themes are not mutually exclusive, they do not sum to 1.0. 

Novices and Experts differed in the frequency with which they voiced most themes. These differences 
will be reviewed in the context of the four Hypotheses followed by an unanticipated result related to response 
time. 


Table 1. Utterance themes, examples, and mean proportions by Novices, Business Experts, and Psychology Experts 


Theme 

Examples 

Novice 

BusExp 

PsyExp 

BECAUSE: Inferences of 
causality 

“don’t know whether they’ve had a change in 
government or if officials have just gotten a lot 
more corrupt but. . .” 

.19 

.41 

.54 

BETWEEN Z: 
Comparisons between z- 
variables 

“In 2008 the big 10 and the emerging economies 
have an equal amount of annual average growth in 
GDP” 

.61 

.83 

.97 

DIRECTION: Within a 
single z - variable 

“Azerbaijan is expected to remain stable ...over 
2010 to 2011, but then is predicted to decrease their 
instability” 

.52 

.58 

.46 

QUANTITATIVE: 
Interval or ratio 
relationship 

“difference increases dramatically in 2007. It is 
maybe 5 times or 4 times greater in 2007 ...” 

.03 

.25 

.39 

TITLE: Repeat title of 
the graph 

“hypnotic susceptibility by field dependence by 
gender” 

.32 

.91 

.92 

TREND: Overall 
direction 

“over a 3-year span, both groups seem to be 
decreasing the number of publications” 

.11 

.18 

.35 

X-AXIS: References to 
abscissa 

“r-axis shows Day 1, Day 2, Day 3” 

.25 

.45 

.36 

K-AXIS: References to 
ordinate 

“The y-axis shows GDP — adjusted GDP — in 
billions of US dollars.” 

.23 

.40 

.28 

Z-Variable: Number or 
name of z- 

“The two lines represent..., respectively, the scores 
for males and for females. . .” 

.18 

.20 

.25 


3.1 Proportion of “Because” Inferences 

Although all participants were asked to “explain the graph as if you were the author and you were explaining 
the results to another person", utterances of the form “variable a causes variable /?” were observed 
infrequently in Novices. A repeated measures 3 (Expertise: novice, business expert, psychology expert) x 2 
(Difficulty: simple, complex) x 2 (Domain: business, psychology) ANOVA revealed only a significant main 
effect of expertise, F( 2, 23) = 4.73, p = .019, r| p 2 = .29. Independent post hoc Tukey tests confirmed that 
psychology experts ( M = .57) attempted more inferences than novices ( M = .18), p = .015; the difference 
between business- and psychology experts was not significant (p = .440), and nor was the difference between 
business experts and novices (p = .214). Hypothesis 1 stating that Experts would provide more inferences 
than Novices was thus supported. 
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3.2 Familiar and Unfamiliar Domains 

Hypothesis 2 stated that Experts would generate more complete explanations of graphs in familiar than 
unfamiliar domains. Excluding Novices, a repeated measures 2 (Expertise, business, psychology) x 2 
(Domain: familiar, unfamiliar) x 2 (Difficulty: simple, complex) ANOVA resulted in only one significant 
effect, the Expertise x Domain interaction, F(l, 14) = 6.56, p = .023, r)p2 = .56. Post hoc t-tests for 
independent samples confirmed that the interaction was due to higher completeness scores for psychology 
Experts on familiar domain graphs (M = .53) compared to the unfamiliar domain (M = .48), t(7) = 3.30, p = 
.013, and business Experts exhibited the opposite effect of significantly lower completeness scores on 
familiar domain graphs (M = .45) compared to the unfamiliar domain (M = .49), t(7) = -2.71, p = .030. Thus 
Hypothesis 2 was partially confirmed by psychology Experts but refuted by business Experts. 

3.3 Nominal, Ordinal, and Metric Conceptual Messages 

The proportion of conceptual messages is the sum of z-variable. Between z-variable, and Quantitative 
proportions (refer to Table 1). A 3 (Expertise: novice, business expert, psychology expert) x 2 (Difficulty: 
simple, complex) ANOVA of the conceptual messages resulted in a significant expertise main effect, F(2, 23) 
= 7.80, p = .003, pp2 = .40. Planned comparisons indicated that business Experts (M = 1.64) generated more 
conceptual messages than Novices (M = 1.11), p = .012, psychology Experts (M = 1.55) generated more than 
Novices, p = .013 but business and psychology Experts did not differ from each other, p = .59 confirming 
Hypothesis 3. 

3.4 Completeness 

A completeness score was calculated by determining the proportion of all nine themes present in each 
participant’s explanation of each graph. The mean completeness scores are shown in Figure 2 for each 
expertise group and for each domain. The Figure suggests that the two Expert groups’ explanations were 
more complete than those of Novices and that this was more pronounced for psychology than for business 
graphs. This was confirmed by a repeated measures 3 (Expertise: novice, business expert, psychology expert) 
x 2 (Difficulty: simple, complex) x 2 (Domain: business, psychology) ANOVA on completeness scores. The 
main effect of expertise was significant, F(2, 23) = 8.02, p =. 002, pp2 = .41 and independent Tukey post hoc 
comparisons confirmed that business Experts (M = .47) provided more complete explanations than Novices 
(M = .27), p = .014, and the same was also true for the psychology Experts (M = .51), p = .004, confirming 
Hypothesis 4. 



Figure 2. Mean completeness scores for expertise and graph domain. (95% confidence intervals were calculated using the 

procedure of Jarmasz & Hollands, 2009) 
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3.5 Total Response Time, Silent Time, Explain Time 

Total Response Time (TRT) for the graph explanation task was composed of Silent Time (ST) plus Explain 
Time (ET). ST was the silent period before participants began their graph explanations and ET was the time 
during which participants voiced their explanations. A repeated measures 3 (Expertise: novice, business 
expert, psychology expert) x 2 (Difficulty: simple, complex) x 2 (Domain: business, psychology) ANOVA 
for TRT revealed no main effect for Expertise (p = .817) nor any significant interactions with Expertise, 
suggesting that the efficiency with which Experts completed the graph explanation task was no better than 
that of Novices. 

A repeated measures 3 (Expertise: novice, business expert, psychology expert) x 2 (Difficulty: simple, 
complex) x 2 (Domain: business, psychology) ANOVA for ET revealed no main effect for Expertise 
(p = .478) nor any significant interactions with Expertise. 

However, a repeated-measures 3 (Expertise: novice, business expert, psychology expert) x 2 (Difficulty: 
simple, complex) x 2 (Domain: business, psychology) ANOVA on ST revealed a significant main effect of 
expertise, F(2, 23) = 7.71, p = .003, pp2 = .41. Independent Tukey post hoc tests confirmed that novices had 
longer silent periods before beginning their explanations (M = 8.95 s) than business (M = 2.17 s,), p = .003, 
or psychology Experts, (M = 3.37 s), p = .016. If ST represents the time required to select and/or initiate a 
strategy then Experts required less time to select their graph explanation strategies than Novices. Novices 
appeared uncertain about what to say or perhaps how to start their graph explanations. 

3.6 Results Summary 

The current research demonstrated a difference between Novices and Experts in their graph exploration in 
terms of the proportion of time Experts attempted inferences in their interpretation of the graph data; and the 
completeness of their explanations. The greater Silent Time of Novices before initiating their explanations 
suggests that undergraduate students struggle with an appropriate strategy to attempt their efforts and the 
results suggest a parsimonious extension to the graph comprehension model of Pinker (1990). However, the 
lack of a consistent effect of familiar versus unfamiliar domain in the performance of Experts leaves some 
question as to the locus of these effects — whether they are evidence of a general expertise effect or one 
limited to a specific domain. These results are summarized in Table 2. 


Table 2. Research hypotheses, results, and conclusions 


Hypotheses 

Results 

Conclusions 

HI. Experts will generate more 

• 

More “because” inferences by BusExp & 

Expertise effect in graph 

“because” inferences than novices 

• 

PsyExp than Novices 

Similar number of “because” inferences by 
BusExp & PsyExp 

exploration supported 

H2. Experts will provide more 

• 

PsyExp psych domain explanations more 

Domain-specificity of 

complete graph explanations for 


complete than business domain 

graph expertise partially 

familiar compared to unfamiliar 
domain graphs. 

• 

BusExp business domain explanations 
similar completeness scores to psych 
domain 

supported 

H3. Experts will generate more 

• 

BusExp and PsyExp generated more 

Supports extension of 

conceptual messages than Novices. 


conceptual messages than Novices 

Pinker (1990) model 

H4. Experts will generate more 

• 

Higher completeness scores by Experts 

Supports perspective on 

complete explanations than novices 


than Novices 

graph exploration where 
completeness=expertise 

Unanticipated 

• 

Silent Time greater for Novices 

Suggests that 


• 

Explain Time similar for all groups 

Expert/Novice differences 
may be due to conscious 
strategy 
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4. CONCLUSIONS 

The present research contributed to an understanding of graph exploration in three ways. First, the 
experiment is among the first to demonstrate an expertise “effect” in the domain of graph exploration. 
Although others have studied graph expertise (e.g., Roth, 2004; Roth & Bowen, 2003) they have not directly 
contrasted novice and expert performance. Previous attempts to distinguish novice and expert graph 
comprehension (Shah & Carpenter, 1995) found no differences between the two types of participants. 
However, since the effect of domain was inconclusive in the current research, it remains unknown whether 
this expertise effect is general or limited to specific domains. 

Second, this experiment showed that Experts adopt a graph exploration strategy in which specific 
elements of a graph are explored. It is proposed that these elements represent a list of conceptual questions 
that is the embodiment of a graph exploration strategy. The addition of this top-down process adds clarity to 
Pinker’s (1990) model of graph comprehension by introducing a mechanism for the operation of expertise. In 
contrast, novices’ strategies were inconsistent. As a consequence, it took them longer to initiate their graph 
exploration, and their explanations were less complete than those of the experts. 

Finally, the issues identified here in Expert/Novice differences in graph explanation lend themselves to 
intriguing ideas in education and data visualization. Perhaps it would be possible to address these to improve 
the data literacy of children or older students (e.g. Feldon et al., 2010), or in the teaching of statistics (e.g. 
Cleveland, 1987; Huff, 1954). In particular, it is reasonable to believe that an instantiation of the Expert 
graph exploration strategies determined here might be embodied in a training regimen to bootstrap the 
understanding of data by Novices. This is research that we have currently underway. It is also easy to 
imagine these reflected in computer-based data visualization tools (e.g. Heer, et al., 2010; Konold, 2007; 
http://datavisualization.ch/tools/). 

Unfortunately, the data are insufficient to determine if the inconsistent effect of domain provides evidence 
of a global expertise effect or if they are limited to specific domains. Perhaps more complex graphs, in terms 
of either visual or semantic complexity would have resulted in more definitive evidence. A replication of the 
current research using interactive graphs might be particularly informative. 

In conclusion, the importance of this line of research is underscored by regular national comparisons of 
student performance in mathematics (e.g. OECD, 2014). The OECD Programme for International Student 
Assessment asserts that the application of mathematics (including graph exploration) is a key attribute of 
“What is important for citizens to know and be able to do?” (OECD, 2014 p. 3). The current research may 
contribute to an improvement in what students can do with data. 
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ABSTRACT 

Assessment of groups in CSCL context is a challenging task fraught with many confounding factors collected and 
measured. Previous documented studies are by and large summative in nature and some process-oriented methods 
require time -intensive coding of qualitative data. This study attempts to resolve these problems for teachers to assess 
groups and give timely feedback. We first operationalize activity theory to holistically frame group work by breaking it 
down into six dimensions. The captured log data by the collaborative software is mapped with these dimensions and as 
a result, six measures are generated with a semantic background. Next, we employ a relatively new clustering algorithm 
- spectral clustering - to categorize groups with similar behaviors, which not only allows us to consider the six 
indicators simultaneously but also has the capability to deal with large online context. The spectral clustering result is 
compared with traditional algorithms and demonstrates better assessment accuracy. Furthermore, since the whole 
process is automated and the group performance indicators are grounded in a meaningful backdrop, it enables teachers 
to offer concrete and personalized help in a real-time format. Theoretical and practical implications are then discussed. 


KEYWORDS 

Assessment, CSCL, Clustering, Small Group Learning, Activity Theory 


1. INTRODUCTON 

Assessment is not only important for evaluating learning outcomes: it may also be a motivating factor for 
students who have a performance goal orientation (Dennen, 08). While the measurement and assessment of 
learning is a major responsibility for teachers, it is also a demanding experience for them due to the heavy 
workload and time-consuming of the assessment activities, especially when learning take place in 
technology mediated group format (Strijbos, 11; Gress et al., 10). Assessment of group learning in 
computer-supported environments is more than the measurement of outcomes; the quality of collaborative 
learning processes is also salient (Strijbos, 11). 

However, process-oriented assessment for group learning in the socio-technical context is a difficult 
problem to resolve in that learning in such an environment takes place through the complex processes and 
interactions of numerous factors, artifacts, and environments (Barab et al., 01). Because of this complexity, 
assessment of group remains by-and-large summative in nature (Gress et al., 2010) where the performance 
of each group was measured by the quality of the solution or product generated. This type of group 
assessment centers on the intellectual product of the learning process rather than the process itself ( Kumar 
et al., 10), overpassing group dynamics, interaction, and the technology-mediated processes. To address the 
complex interactions of small groups, some assessment with process-orientation endeavors require 
integration across multiple coding schemes and sources of data (Hmelo-Silver et al., 11). These assessment 
efforts often rely heavily on conventional methods such as content analysis, coding of observations, 
interaction analysis, etc., and are therefore very time-intensive, adding further burdens to the already heavy 
duties of teachers. Further, most documented studies on group assessment, both summative in nature or 
towards collaborative process, were conducted after collaboration (Gress et al., 10), and hence are lack of 
the capability to provide information in a timely format for teachers to act upon. 


47 



ISBN: 978-989-8533-23-4 ©2014IADIS 


We describe a methodology that attempts to assess group as the unit in CSCL in order to provide 
actionable information to teachers in a timely manner. Specifically, this work first explores operationalizing 
activity theory to holistically frame group activity in a CSCL context by breaking down group work into six 
dimensions. Then, rather than performing observations or content analysis to produce our measures, 
electronic trace data generated by the collaborative software are mapped to the six dimensions for measure 
construction. This step not only lays the groundwork for automating the group assessment, but also 
contextualizes the data with a semantic grounding. Next, we move beyond identification and analysis of 
those measures to infer group learning using human judgment but employ a relatively new clustering 
algorithm to categorize groups with similar behaviors into clusters, which allows us to consider the six 
indicators simultaneously as well as deal with large online context for offering concrete advice to students. 
This paper is organized as follows: We first describe the theoretical framework that guides this study. Then 
the research background is described. Next, methodology, and results are presented. Last, we discuss the 
implications of our findings for this study as well as pointing out future research directions. 


2. THEORETICAL FRAMEWORK 


Tools 


Tools. Systems. Environments 



[Explicit & Implicit Rules Direct & Indirect Communication Balanced Contribution 

Figure 1 . Activity theory in group context 


Measure- 

metric 

Object 


Table 1. Description of activity theory operationalization in CSCL 


Definition 


Complete learning tasks collaboratively such as solving a problem or producing an artifact 


Subject Individual student involves and participates in problem solving towards the Object. 


Tools 

Community 

Rules 

D. of Labor 


Computers, tools, systems, and environments that mediate the learning and collaboration activity. 
Direct and indirect communication enables the group of students to maintain a sense of community 
and belonging. 

Implicit and explicit rules and guidelines that constrain the activity. 

All group members make concrete and balanced contributions to the overall object. 


The Activity System model developed by Engestrom (1987) offers a way to comprehensively frame the 
collaborative knowledge development process while linking together social behavior and its 
interdependencies (de Laat, 2006). Engestrom’s (1987) activity model includes six interacting components: 
subjects, tools, objects, rules, community and division of labor (see Figure. 1). Contradictory tensions 
between the elements of the Activity System serve to produce the outcome of the activity. The activity of 
learning (Basharina, 2007) is “the joint activity of a student, physical/symbolic tool(s), and another 
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person(s) performing together as a working social system to achieve some outcome under constraints such 
as rules.” In our CSCL group assessment context, the outcome and process of this transformation may both 
be seen as learning and knowledge. It is the sum of the system components and the tensions among them 
that make up the learning and influence the learning outcomes. Activity Theory helps us to address the 
complex interactions and see into group performance in a socio-technical CSCL environment (see Table 1). 


3. RESEARCH CONTEXT 


3.1 Virtual Math Teams 

In this study, we operationalize activity theory as a lens for making sense of electronic trace data from a 
synchronous math tool, Virtual Math Teams with Geogebra (VMTwG) software (Figure. 2). We focused on 
several modules of a course include teams of three to five practicing teachers who were learning how to 
implement the curriculum in VMTwG while going through the curriculum themselves. There were 18 
groups in total in this course using the VMTwG software. The five modules we analyzed included: 
“Constructing Dynamic-Geometry Objects”, “Exploring Triangles”, “Creating Construction Tools”, 
“Constructing Triangles”, and “Inscribing Polygons”. The frill curriculum currently includes a total of 21 
topics, and is available at the project website (http://vmt.mathforum.org). 

Figure. 2 provides us with a visual guide for understanding the cognitive learning discourse in VMT. 
There are four sections in Figure. 2. Section A reveals the time dimension: the VMT replayer bar. Each 
action within VMTwG is logged with a timestamp. Section B is the chat window. Here, text is entered in 
order to chat with others in a group. Future analytics in this project will focus on the automated analysis of 
the actual text in those windows, in concert with GeoGebra gestures. Sections C and D are related to 
Geogebra actions. C is the “Take Control” button. Section D is the GeoGebra window itself. Here, students 
work to create an equilateral triangle within an equilateral triangle using multiple approaches. This is an 
ordinary part of how VMTwG facilitates interactive problem solving discourse among students. On the 
other hand, the design of the VMTwG, as a whole, places teams into separate learning spaces for small 
group interaction. As a result, teachers have difficulty facilitating or directing activity in a timely manner; 
with 3 or 4 students per group, 18 groups doing an hour of activity translates into 18 hours of replay. 



Figure 2. VMTwG of an analytical tool for collaborative math discourse 
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3.2 Dataset Description 

We collected log data in .txt format from five modules in a single course. The data centers on specific event 
types from the CSCL environment (VMT): Chat, Awareness, Geogebra, System, and WhiteBoard (Wb). 
The Chat event type logs all the messages that students communicate with each other. Awareness records 
the actions of erasing the chat messages when the student realizes they are full on the chat bar. Geogebra 
logs information on how students visually construct a geometry artifact (e.g. add a point, update a segment, 
etc.). The System event type records information on how the VMT environment is accessed. 


4. METHODOLOGY 

Based on the trace data captured in the VMT logs, several quantitative measures are developed according to 
the components of the Activity System as shown in Figure 1 . Further details regarding measure construction 
based on Activity Theory can refer our pervious paper (Withhold for Review). This process of measure 
construction allows for the development of our automated CSCL assessment technique, while at the same 
time providing a view of the log data that is intelligible for teachers. 

4.1 Measure Construction 

4.1.1 Subject 

Subject in Activity Theory represents individual students' efforts in solving a problem. As a reflection of 
individual effort in a group, we consider whether they perform equally in the modules. Therefore, we 
calculate the frequency of participation under individual tasks and then use the value of standard deviation 
of these values to represent Subject. 

4.1.2 Rules 

Rules include implicit and explicit rules. Students have to perform actions that the VMT environment 
offers. We count the number of distinct functions the group of students used to represent the Rules. 

4.1.3 Tools 

Under the VMT context, the tools are the System and Wb, where the groups’ actions for tool usage are 
registered. Hence, the Tools dimension for each group is the summation of the actions in these two event 
types. 

4.1.4 Community 

All the communications that help maintain the community structure. Community can be presented as the 
summation value of the frequency of chat messages. 

4.1.5 Division of Labor 

Division of Labor is a measure of how the balance of the workload is shared among team members. This 
dimension would have the highest value if all the members in a group shared the workload equally and 
would have the lowest value if just one of the members took care of the whole problem. Thus, the balance of 
the work among team members is based on the function group effort (Geogebra dimension) with the perfect 
division 1/N, where N is the number of students in that Group. 

4.1.6 Object 

The CSCL activity is to achieve the object of a group of students actively participating in the whole class to 
solve problems. Object is a function of number of modules involved, event types used, and totally frequency 
of participation. 

In sum, relying on activity theory, we build a quantified model for group performance in CSCL activities 
specific to the VMT environment: [Subject, Rules, Tools, Community, Division of Labor, Object]. In 
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addition to providing a principled way for measure selection and construction, these theory-grounded 
measures have a semantic background to facilitate teachers to understand. 

4.2 Spectral Clustering 

An activity system is characterized by the internal tensions among its components. The tensions are the 
moving force behind disturbances and innovations and eventually drive the system to change and develop, 
in this context toward an outcome of group learning. Therefore, it is hard to compute one value as functions 
of the six dimensions to indicate the learning or performance result of a group of students, especially 
considering the complexity of the nature of learning. In this exploratory study, we investigated a novel 
clustering algorithm - spectral clustering - to place groups with similar behavior patterns in the same 
cluster. Spectral clustering brings into consideration of all the six dimensions in the activity system rather 
than accounting for only one dimension. Also, it allows teachers to give similar advice to a number of 
groups at a time, which might be particularly helpful in large-scale online context. 

On the other hand, compared with traditional clustering techniques (K-means, EM, etc.), which depend 
on distances from cluster prototypes usually assuming Gaussian distributions, spectral clustering does not 
make strong assumptions on the statistics of the clusters but bases on the eigenstructure of an affinity matrix 
to partition the data objects into disjoint clusters (Von Luxburg, 2007). This feature is important because 
educational data almost always contains exceptional students and groups (outliers) who can achieve a good 
performance with little effort or fail without any sensible reason. The cluster statistics does not necessarily 
follows a certain distribution. Further, empirically, spectral clustering has produced better results than 
traditional algorithms (Galluccio et al., 2013; Von Luxburg, 2007). 


5. RESULTS 
5.1 Measures 


Table 2. Sample activity theory informed group standardized performance measures 


Group 

Dimension 

Subject 

Tools 

Community 

Rules 

Division of 
Labor 

Object 

Group 1 

0.576 

-0.231 

-1.066 

0.903 

0.169 

-0.497 

Group 2 

0.353 

-0.376 

-0.766 

0.721 

-0.121 

-0.666 

Group 3 

-0.463 

-0.767 

0.647 

-1.312 

0.599 

0.724 

Group 4 

1.550 

-0.965 

-1.089 

1.028 

2.091 

2.091 

Group 5 

-1.469 

2.247 

1.644 

-0.550 

-0.300 

-0.376 









Group performance is represented as 6 dimension sets in Table 2 (after standardization). There are 18 
groups in total. By investigating those numbers alone, the teachers are already able to provide specific 
actionable advice to a particular group. For example, if the value of a group in the Community dimension 
(Group 4) is very low, the teacher could suggest the group to communicate more between team members. 
Similarly, if the group has a low value on Division of Labor (Group 5), then the teacher can tell that the 
contribution from the group members is not equal. Activity theory equips us with a holistic way to describe 
groups’ participation performance in a CSCL environment rather than via ad-hoc guesswork. These 
quantified results provide semantic clues that instructors may use to investigate the group work. However, 
simply examining and comparing those numbers is not a valid or reliable way to assess the group 
performance. Neither can this method scale up to large online context when there are tens and even 
hundreds of groups. Therefore, the clustering method from the next section is a good complement to fill 
these voids. 
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5.2 Cluster Results 

5.2.1 Clustering Performance 

Clustering algorithm is performed on the constructed 6-dimension measures informed by Activity Theory. 
To verify the effectiveness of the clustering result, we used a subjective metric, where the ground truth data 
is labeled by human evaluators. Specifically, the group performance is judged both by collaborative products 
and key actions and words used frequency, in which keywords and actions are chosen relying on the authors 
five years of experience of participation in the VMT development and analysis of small group learning. The 
list of key actions is: move graphics view, polygon, segment, perpendicular, circle, insert, and triangle, 
zoom in. erase, join the room. The list of key words and symbols are: right triangle, angle, triangle, 
equilateral, perpendicular, make, point, control, create, move, drag, scalene, all capital words, ?, !. Then a 
1-12 mark is assigned to each student, where 12 is the highest. Then under the circumstance of dataset 
known classification, a common indicator to assess the clustering results is rand index (RI), a measure that 
is to test the consistence degree between the clustering results and data external standard classes (Krieger & 
Green, 1999). The accuracy of rand index is equal to the ratio of the correct matching-pair number to the 
total of matching-pair number. 

#(the correct matching pair) 

RI = 

#( the total matching pair) 

To further validate the proposed spectral clustering method, various cluster number k are chosen k = 2, 
3, 4 and also compared the obtained accuracy with the baseline algorithms, k-means and HAC. 
Accordingly, the ground truth group data are placed into different groups by the concept of percentile in 
response to different k. For example, if the cluster k is set as 4, then based on the value of the mark labeled, 
students with 1-3 marks are placed into one group and then 4-6, 7-9 and 10-12 into different groups. 
Spectral and k-means methods obtained different clusters by directly set the cluster number in the coding 
process, while HAC got different cluster by choosing different cut standards. Furthermore, with different 
parameter settings and initializations, 10 different runs are conducted on each algorithm by clustering the 
constructed 6-diemison measures informed by activity theory. Comparing the generated clusters with the 
human labeled results, the average values of RI obtained from various algorithms are displayed in Table 3. 

To illustrate, the described spectral clustering outperforms the baseline k-means and HAC models in all 
the conditions of different cluster numbers. It has the best performance when choose cluster k = 2 and 
reaches 89.4% accuracy. 


Table 3. Clustering Performances 


Rand Index 

Spectral 

Clustering 

k-means 

HAC 

k = 2 

89.4% 

83.9% 

68.3% 

k = 3 

85.0% 

81.1% 

78.3% 

k = 4 

80.6% 

69.4% 

56.7% 


5.2.2 Cluster Interpretation 

In combination with the contextualized measures informed by activity theory, cluster analysis is not only 
able to facilitate scaling up in online context, but also offer actionable intelligence for the stakeholders. We 
presented one of the experiments using spectral cluster analysis as an example in Table 4 when we set the 

cluster number Table 4 shows the standardized means of each cluster over the six dimensions and 

number of groups in each cluster as well as the range of each dimension. To illustrate the actionable power. 
Cluster 1 is chosen to demonstrate. When the teacher looks into this table, he or she realize that Cluster 1 
has 5 members in it, which tend to have best performances in Tools (0.61), Community (0.46), Rules (0.87) 
and Object (0.29) dimensions but have lowest ranking in Subject (-0.57) and Division of Labor (-0.45). The 
low ranking Division of Labor reflects that the members in these groups do not make the same contribution 
and effort to problem solving. In fact, low value on the Subject aspect is a further proof implying that some 
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of the students in the groups are very active but some are not. The teacher therefore can log into the rooms 
where those groups are and identify the specific students and encourage balanced contribution to the tasks. 
Or if in the context of a large number of online groups, the teacher can send one reminder or report to those 
groups at the same time to remind their problems. While this is from the teacher’s perspective, the table is 
also easily able to present to the students themselves and improve their self-awareness of their performance 
status. 


Table 4. Sample Spectral Cluster k = 3 Results 


Cluster Size 

Cluster 1 

Cluster2 

Cluster3 

5/18 

7/18 

6/18 

Measure 

Range 

(Standardized) 

Cluster Means 

Subject 

[-1.47, 2.87] 

-0.57 

0.45 

0.02 

Tools 

[-1.19, 2.03] 

0.61 

-0.15 

-0.31 

Community 

[-1.88, 2.09] 

0.46 

-0.33 

-0.33 

Rules 

[-1.42, 2.25] 

0.87 

-0.15 

-0.75 

D. of Labor 

[-2.59, 1.10] 

-0.45 

0.79 

-0.35 

Object 

[-1.67, 2.09] 

0.29 

0.15 

0.08 


6. DISCUSSION 

The group is the fundamental unit in CSCL. Many studies of group learning use summative assessment 
methods (e.g. final solution, grade) to measure performance. These assessment approaches usually overlook 
the collaborative process and the affordances of technology in contribution to group learning (Barab et al., 
2001). Qualitative studies e.g. content analysis (Arnold et al., 2009), and conversation analysis (Safin et al., 
2010) usually are very time intensive and therefore impractical for teachers to implement. Many 
quantitative explorations and data mining algorithms usually based on ad-hoc guess work to build their 
measures and do not address the complex small group dynamics and interaction systematically (Mirriahi et 
al., 2013). In addition, most of the assessments are conducted after the collaboration and therefore unable to 
support real-time intervention and reflection. We attempt to solve those problems by designing an 
automated group assessment methodology that mining the electronic trace data to provide actionable 
information to teachers in a timely format. 

There are multiple perspectives on assessment in the CSCL literature related to group learning e.g. 
group cognition (Stahl, 2006), learning as participation (Sfard, 1998), and knowledge creation (Lipponen, 
Hakkarainen & Paavola, 2004). In this paper we zoomed out and explored a different way of developing 
assessment that is coherent with theory and connected with an advanced algorithm. Activity theory 
holistically frames group participation in our framework, which addresses situatedness, contextuality, and 
social mediation. From a practical perspective, there is an overreliance on text-based measures to assess 
learning in CSCL (Gress et al., 2010). Coding of the discussions and content are usually quite time- 
consuming. It is extremely difficult for a real world teacher to code and provide timely feedback to students. 
In addition, these coding methods and frameworks are not always shared (Gress et al., 2010) which leads to 
difficulty in maintaining consistency among different evaluators. The proposed method, from measure 
construction to clustering is totally automated. Therefore, it could significantly reduce teachers’ assessment 
burden and has the affordance for teachers to provide timely feedback. The automated method can also 
increase the consistency of evaluations and improve the reliability of the results. 


7. CONCLUSION 

This study presents an activity theory-grounded spectral clustering model for assessing groups in a 
technology-mediated environment. We first operationalized activity theory to holistically quantify the group 
participation in VMT environment and generated 6 measures. Then, the spectral clustering algorithm is 
coded and the assessment accuracy is compared with traditional techniques, which the proposed spectral 
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clustering achieved the best accuracy. Since the whole process is automated and the generated group 
performance indicators are with a meaningful semantic background, it allows teachers to provide concrete 
and personalized feedback in a real-time manner. A combination of qualitative and quantitative assessment 
measures may offer the best way to assess learning and performance in a CSCL environment. However, 
qualitative assessment is often time consuming. Future work can investigate the technique of natural 
language processing of the chat log data and incorporate the results into the activity theory measure 
construction system in order to further inform group assessment in CSCL. 
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ABSTRACT 

This paper explores how wiki may be used to support secondary education students’ collaborative creativity processes 
and how such interaction can promote critical and creativity thinking. A science case-based project in which 81 
secondary students participated was designed, implemented and evaluated. Students worked in the science wiki project 
during two weeks. We scaffold students to be collaboratively engaged in purposeful critical and creative discourse in 
order to solve collectively science challenges and construct meaning about topics related to environmental challenges. 
Through the analyses of students’ contributions in the wiki we have characterized collaborative creativity processes in 
science inquiry that includes performance (processes to develop a novel way of approaching and understanding the 
problem) and collaboration (peer collaboration, dialogue). The significance of the paper relays on the operationalization 
of the collaborative creativity processes in the wiki within four overarching learning to learn together skills, which are: 
distributed leadership, mutual engagement, peer evaluation and group reflection. 

Our findings showed that the wiki environment afforded the development of an effective and creative online collaborative 
learning community. In student’s wiki contributions, the four learning to learn together skills took place. However, not all 
the groups displayed the four learning together skills during their collaboration in the wiki and there were differences 
among groups in relation to the presence and proportion of these skills. We discuss the contribution of these four 
learning to learn together skills for the collaborative creativity processes and the relation of the presence of the above 
mentioned skills with the level of creativity showed in the collaborative writing product students produced in the wiki 
project. 

Besides, the paper discusses a series of issues that instructors should consider when wikis are incorporated into teaching 
and learning for creativity. We claim that embedded scaffolds to help students to argue and reason creatively in their 
contributions in the wiki environment are needed. 


KEYWORDS 

Wiki, Creativity, Collaboration, thinking skills, secondary education. 


1. INTRODUCTION 

Collaboration is a central tenet of the new Social Web. In Web 2.0 technologies, users are active participants 
who dynamically and collaboratively create new content of new solutions for common problems. In the case 
of wikis, this technology allows participants to create a collective document by editing, discussing, and 
sharing information about a topic of common interest (Wichmann & Rummel, 2013). Educational 
professionals are increasingly interested in understanding the conditions needed for successful collaboration 
in creative web 2.0 endeavors. Etelapelto & Lahti (2008) claim that for successful collaborative creativity 
participants need to build on each other’s ideas in order to reach an understanding that was not available to 
any of the participants initially. Besides, the participants need to be committed to shared goals, and have 
sufficient trust in each other to join in the shared endeavor. Furthermore, the participants must also enter into 
critical and constructive negotiation of each other’s suggestions and well-grounded arguments and counter- 
arguments need to be shared and critically evaluated through collective talk. In this paper, we claim that all of 
these processes included in collaborative processes can be developed using purposely wiki technology. 

Thus, for understanding creativity in collaboration we need to study the dynamic processes of 
co-construction. Some researchers claim that this understanding requires qualitative studies which focus on 
the group processes, group dialog characteristics and how common work unfolds over time. Our research 
falls down in this line of research by analyzing the mechanisms students develop to solve creatively and 
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collaboratively an environmental problem in a wiki. The main aim of our study is characterize the features of 
the learning to learn together skills student develop during wiki-supported group work and that are capable to 
support collaborative creativity in a wiki community. 


2. UNDERSTANDING COLLABORATIVE CREATIVITY 

In the context of knowledge and technological age, most knowledge creation is conducted by teams working 
with and around technology. In these learning situations the development of learning to learn together skills 
(for hence, L2L2) to solve a problem collaboratively and creatively are crucial. This paper will focus on 
understanding the mechanisms that can trigger creativity in collaborative endeavors. We will study some 
collaborative creativity mechanisms in the context of solving a social and complex science problem using a 
web 2.0, a wiki. 

Recent educational research has pointed out the difficulties to define creativity, and empirical researchers 
have employed different operationalization of the term, conceptions of creativity usually agree that creativity 
means novelty and socially valued (John-Steiner, 2000; Sawyer, 2003). 

Sawyer, a leading creativity researcher exemplifies collaborative creativity by a jazz performance which 
requires an entire jazz ensemble; performance emerges from the interactions of four individuals working 
collaboratively. Group’s creativity is greater than the sum of its parts and to understand group creativity, we 
have to focus on the processes of collaboration among members (Sawyer, 2012). This author claims that to 
understand collaborative creativity research is needed form the “process approach” or “mechanism 
approach”. This approach with qualitative research approach examines the group processes, group dialog 
characteristics and how common work unfolds over time. Our research falls down in this line of research by 
analyzing the mechanisms students develop to solve creatively and collaboratively an environmental problem 
in a wiki. Different researchers have already paid attention to some mechanisms that might promote 
collaborative creativity, next we revise some important studies for our research purposes. 

In a learning group discussion to solve a community problem, novelty means that new and alternative 
ideas are suggested in respect of the problem at hand. Nevertheless, the novelty of the idea alone is not 
sufficient for collaborative creativity; in addition the novel idea must be in some way reasonable and sensible 
in the situation concerned. This means that to be creative an idea must be socially appropriate and thus be 
recognized as socially valuable in some way (Sawyer, 2004; Sternberg, 2003). 

Furthermore, Etelapelto and Lahti (2008:227) suggests that in a peer-group learning community neither 
the emerging problems nor their solutions are known in advance, but the group needs to work together in 
order to define the problems and find solutions to them. Thus, creativity in collaboration can be understood to 
emerge within dynamic processes of co-construction; these will produce novel - and appropriate - ideas 
regarding the problems faced in collective learning endeavors. 

Wegerif et al (2010) in a research in which a dialogic perspective of analyzing creative thinking in online 
dialogues was adopted; claims that the creative process in a collaborative learning situation depends more on 
a tension between different perspectives rather than a shared framework. Thus, creative thinking emerge 
when further entails opposing ideas and disagreements being thoroughly discussed, in such a way that 
differing opinions and conceptions are related to each other. In such a process of collective learning an 
elaborated understanding of the learning topic can emerge. 

The communicative and social dimension of collaborative creativity is also highlighted by Sonnenburg 
(2004) theoretical framework for creating in collaboration. Participants have to be mutually engaged in the 
process of communication during the collaborative resolution of a task and they have to present a working 
style distinguished by a serious of dispositions that can favor the emergence of creativity in collaboration. 
This author highlights an open and free communication in which all collaborators have the same chance to 
contribute to the course of performance, and the same right that his contributions are taken seriously. Mutual 
trust and risk-taking are other key dispositions in collaborative creativity. 

Taking into account previous arguments and research in the topic of collaborative creativity we argue that 
the reality of Internet mediated learning and creativity implies to develop learning how to learn together 
(L2L2) with others and this type of learning combines the dimension of task management with the dimension 
of social relationships. Learning to learn together is a complex competence requiring that all the group 
members are able to coordinate, regulate and plan the learning task, balancing issues of individual ability. 
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motivation and expectations through constant dialogue. From this point of view, one main issue is how to 
support our students to create and be engaged in powerful, critical and reflective dialogues using web 2.0 
technologies that help them to co-construct new knowledge and create novel solutions to relevant social 
problems through online interaction with others. 


3. WIKI AFFORD ANCES TO PROMOTE COLLABORATIVE 
CREATIVITY PROCESSES 

Wikis are collaborative and participatory tools. Researchers have already described the broad range of 
potential pedagogical applications for wikis (Lund and Smordal, 2006). 

Wikis have three major functions to facilitate collaborative creativity processes: 

(a) Editing function that supports multiple users to create a common text in which dynamic and creativity 
processes of co-construction the common text may appear. 

(b) History function that records all edits, by means of color coding, allowing users to trace all revisions 
being made. The history log enables edits to be traced to the users. 

(c) Discussion page that enables asynchronous written communication between users by providing 
explanations and posting comments on various issues related to the common text. In our study, we have 
designed a pedagogy model to reinforce the use of wiki functions to promote collaborative creativity 
processes. Thus, the discussion page has been emphasized as a dialogic space in which students should 
propose new ideas, new and different perspectives to face the science problem they should solve together. In 
the discussion page, students were encouraged to argue and reflect their thoughts. 

Although the features mentioned above are characteristics of wiki design that may enhance the 
collaborative processes, it remains unclear which pedagogical approach contributes most to successful 
collaborative learning processes using wikis. Besides, there are still relatively few studies on the use of wikis 
to promote collaborative creativity processes in secondary education. 

In this paper we claim that participation in a wiki for enhancing creativity and collaborative activity 
requires that participants develop learning to learn together skills that combine task management with peer- 
group learning skills. To support this claim, we designed, implemented and evaluated a science project in 
which secondary students used a wiki environment, with the specific aim of establishing and supporting 
collaborative creativity interaction, while engaging in a collaborative writing task. The aim of our research 
study is characterize the features of these learning to learn together skills capable to support collaborative 
creativity in a wiki community. 

With our study, we hope to contribute to the discussion about the pedagogical parameters that need to be 
considered in the design of Web 2.0 supported collaborative learning environments in Secondary Education, 
in order to support students to open up, widen and deepen dialogic spaces for thinking and creating new 
solution together, in the new global communication era. 


4. OBJECTIVES AND RESEARCH QUESTIONS 

In this study we aim to operationalize creative collaboration in a wiki taking into consideration the aspects of 
collaboration that combines the dimension of task management with the dimension of social relationships. 
These aspects have been included in the next four overarching learning to learn together skills: 

a) Mutual engagement 

Mutual engagement ensures the coherence of a community over time and is an essential component 
of any practice. Shared object, artefacts or content provide a rich repertoire of referential anchors for 
mutual engagement and understanding. 

In our wiki project, the next four features are included in this skill: i) joint attention to the ideas and 
contents written in the wiki common text; ii) mutual and shared participation with the ideas proposed 
in the wiki text; iii) shared opinions and iv) explicit support (see Figure 1). 

Our pedagogical model emphasizes the use of the discussion page of the wiki as a dialogic space in 
which students propose, argue and make explicit new ideas and perspectives. For this end, students 
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were taught in the use of specific thinking together language and values in order to make more 
feasible that creative thinking could emerge. 

b) Distributed leadership 

We view leadership as a reciprocal social process instead of the property of an individual, 
leadership responsibilities are shared within the group, and there may be no sharp boundary between 
leaders and followers. All students should be able to constantly negotiate the distribution of 
leadership according to situational and social change. This awareness of distributed leadership 
around particular topics breaks down dominating coalitions, hierarchical relationships, social 
exclusion and isolation. 

In our wiki project, the next three features are included in this skill: i) students’ explicitly taken 
responsibility of some aspects of problem solving ii) explicit organization of group work and iii) role 
distribution (see Figure 1) 

c) Peer assessment 

Evaluation of the ideas and contents proposes by their peers and the products proposed by the 
different members of the group. Our pedagogical model emphasizes the use of the discussion page of 
the wiki as a dialogic space in which opinion and evaluation of others’ ideas should emerge. For this 
end, students were taught in the use of specific thinking together language and values in order to give 
students’ support to give opinion and evaluation to others’ ideas both in the discussion and in the 
editing pages of the wiki. 

In our wiki project, the next two features are included in the peer assessment skill: i) make 
explicit individual differences in terms of different points of view, contradictions about how to solve 
the science problem; and ii) evaluation of the product or the solution (or partial solution) of the 
problem. 

d) Group reflection 

As a shared object, a representation of a group learning process constantly evolves and students’ 
shared understanding of the object can be considered as a process of knowing. To make this process 
of knowing explicit to the group, we identified three distinctive orientations for group reflection in 
the wiki project: 

i) Reflecting on problem solving processes; ii) Reflecting on learning atmosphere, emerging 
roles, norms and gaps between individual and collective outcomes; and iii) group regulation 
processes (see Figure 1) 

The Figure 1 present the representative features of each skill and short example extracted from students’ 
collaboration in the wiki 

( L2L2 CODING SCHEME 


To improve our Final writing, 
we ask you to 
check the spelling 

Two of us agree in the process 
we have done so far 


We have to negotiate the . — | Group organization 
final text we want to write 



We could use this text 
structure for the final Text 



Learning 
to learn 
together 
(L2L2) 


| Mutually participation 
I Sharing opinion 
I Explicit support 


I would not like paying for energy, 
because the heating plant would produce it. 


People can have health problems because 
of the pollution from the heating plant. 


f We don't agree on.... | 


Individual differences 


Product analysis 


/■ ■> 
We have writen about the 
consequences we should consider 

/ 


Do you think that would happen 
if the heating plant 
was underground? 



Figure 1 . Features of collaborative creativity skills 
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4.1 Research Questions 

The research questions of the study can be specified as follows: 

1. What features of students’ collaboration in a wiki can be operationalized as: 

a) Mutual engagement 

b) Distributed leadership 

c) Peer assessment 

d) Group reflection 

2. Can we find a relationship between the presence of the four L2L2 and the collaborate creativity 
solution of the science problem and showed in the collaborative writing product students produced in the 
wiki project? 

4.2 Method 

A case study was designed in which eighty-one secondary students participated in this study (13-14 years 
old). Students worked together in pairs and in groups of 6 students, first at a computer-based science task 
(Webquest). The wiki environment was used to create a joint informative text about the science topic, 
together with two other pairs. 

4.2.1 The Task 

The students participated in a science project, spanning 8 one-hour lessons, which were divided into three 
different phases with distinctive learning objectives. The first phase was two-hour lessons with the specific 
aim to prepare students to collaborate in the wiki environment and to enhance their collaboration process 
using a “Thinking Together” approach (Dawes, Mercer & Wegerif, 2000). Students were taught to improve 
their discussions in the wiki by using sentence openers as a scaffold to enrich and diverse students’ 
contributions in the wiki. Students were taught about five kinds of openers: 1) giving information (e.g., in my 
opinion); 2) asking for someone else’s point of view (e.g. What do you think about; could you give an 
example) 3) expressing disagreement (e.g., I do not agree with; because); 4) expressing agreement (e.g., I 
agree with; because) 5) give reasons and summaries the discussions (e.g., to synthesize; we think; so). 

In the second phase, during the next three class sessions, students researched the topic they would write 
about later, i.e., environmental issues about the construction of a heating plant next to your city. Working in 
pairs, the students undertook a web-based inquiry activity, which was a new topic for them. In the activity, 
students had to search, select, integrate and argue about different types of information on the web about 
heating plant. At the end of this stage, each pair wrote an initial propositional text in which they had to 
present their ideas about pros and cons of construction a heating plant next to their city. 

In the third phase of the project, three pairs of students were grouped together in the wiki environment in 
order to write a collaborative text about students’ arguments in favor or against the construction a heating 
plant next to their city. 

4.2.2 Data Collection and Analysis 

In order to inform about our first research question - what features of students’ collaboration in a wiki can be 
operationalized as: mutual engagement, distributed leadership, peer assessment and group reflection- we 
carried out detailed content analyses of the students’ contributions in the wiki. A coding scheme was used to 
characterize students’ contributions. The coding scheme is presented in Figure 1. 

This coding scheme has been built by revising the educational research made in the area of individual 
meta-learning promoted by computer-supported collaborative learning (e.g. James, 2006) and the revision of 
research studies in which enabling and scaffolding group work supported by technology can promote 
learning processes where students move from an individual process to a process where students can support 
each other’s intellectual engagement by sharing knowledge, learning through interaction and co -construction 
knowledge (e.g. Stahl, Koschmann & Suthers, 2006; Scardamalia & Bereiter, 2006) 

The coding process consisted of two steps: a) dividing the wiki contribution into meaningful units, and b) 
assigning a code to each unit. We decided to segment the notes into units of meaning by using semantic 
features such as ideas, argument chains, and discussion topics, or by regulating activities such as making a 
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plan, asking for an explanation, or explaining unclear information. Validity and reliability aspects were 
considered in the study. 

In order to inform our second research question - can we find a relationship between the presence of the 
four L2L2 and the collaborative creativity solution of the science problem and showed in the collaborative 
writing product students produced in the wiki project?, a rubric for creative writing texts was elaborated. 

Two investigators used the rubric to rate 25% of the texts and after solving minor disagreements, the 
rubric was finally created. 

In order to determine the level of creativity showed in the collaborative text, the proposed rubric contains 
4 different levels marked from 1 to 4. The 4 levels are described below: 

- First level, “listing information". Texts in which students do not solve the problem or contribute 
anything new, do not come to a conclusion, do not define whether they are for or against and offer no 
solution. They just made a list / text with factual information about the problem. 

- Second level, “Divergence Processes”. In these texts students take into account others’ information 
about the problem. Besides students present and discuss different points of view, however students do not 
reach a conclusion or provide a solution for the problem. 

- Third level, “Convergence Process”. Texts in which students besides to present and discuss about each 
other’s point of view, students converge in a conclusion and solution for the problem.. 

- Fourth level, “Creative solutions”, in these texts besides the presentation of divergence and convergence 
processes, students present a brand new, original and realistic solution for the problem. Students reach this 
creative solution consulting and developing new information or new ideas. 

4.2.3 Results and Discussion 

The preliminary results of our study are presented in Table 1. Our findings showed that the wiki environment 
afforded the development of an effective and creative online collaborative learning community. Students 
write an average of 10 contribution per group and were actively engaged in discussing and writing the 
common text. 

Besides, eleven out of the fifteen groups presented the four overarching learning to learn together skills 
proposed in our study, which are: distributed leadership, mutual engagement, peer evaluation and group 
reflection. Therefore, we can conclude that the proposed characterization of collaborative creativity processes 
may be useful for understanding how a group orchestrates their collaboration processes in order to find a 
novel solution for a science problem. 


Table 1 . Results in the different variables studied in the research 


Group 

Number of 
wiki 

contributions 

Numbers of 
meaningful 
units 

Creativity in 
the wiki 
collaborative 
text 

L2L2 skills (in percentage) 





Leadership 

11% 

Peer-assessment 

22% 

1 

10 

62 

4 

Engagement 

46% 

Group reflection 

21% 





Leadership 

18% 

Peer-assessment 

55% 

2 

5 

12 

3 

Engagement 

0 

Group reflection 

27% 





Leadership 

11% 

Peer-assessment 

17% 

3 

10 

85 

2 

Engagement 

50% 

Group reflection 

22% 





Leadership 

12% 

Peer-assessment 

23% 

4 

3 

17 

2 

Engagement 

65% 

Group reflection 

0 





Leadership 

20% 

Peer-assessment 

14% 

5 

9 

72 

3 

Engagement 

36% 

Group reflection 

30% 





Leadership 

5% 

Peer-assessment 

32% 

6 

3 

19 

2 

Engagement 

53% 

Group reflection 

10% 





Leadership 

6% 

Peer-assessment 

38% 

7 

3 

33 

2 

Engagement 

53% 

Group reflection 

3% 
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Leadership 

18% 

Peer-assessment 

56% 

8 

9 

28 

1 

Engagement 

19% 

Group reflection 

7% 





Leadership 

0 

Peer-assessment 

13% 

9 

7 

32 

3 

Engagement 

87% 

Group reflection 

0 





Leadership 

0 

Peer-assessment 

26% 

10 

8 

27 

0 

Engagement 

65% 

Group reflection 

9% 





Leadership 

13% 

Peer-assessment 

42% 

11 

13 

114 

3 

Engagement 

37% 

Group reflection 

18% 





Leadership 

19% 

Peer-assessment 

37% 

12 

19 

54 

3 

Engagement 

31% 

Group reflection 

13% 





Leadership 

15% 

Peer-assessment 

26% 

13 

22 

103 

2 

Engagement 

33% 

Group reflection 

26% 





Leadership 

4% 

Peer-assessment 

19% 

14 

10 

47 

2 

Engagement 

75% 

Group reflection 

2% 





Leadership 

27% 

Peer-assessment 

31% 

15 

16 

117 

3 

Engagement 

33% 

Group reflection 

9% 


Features of the four skills are displayed in the students’ wiki contributions. In table 2 an example of a 
wiki contribution is presented, it belongs to a contribution made in the wiki negotiation page. It can be seen 
how students move smoothly among the features of the four L2L2 skills. Besides, it can be seen how the wiki 
function that let students to share different digital objects (initial pair proposal for the text, negotiation page 
and writing page) in which students make their ideas and argument explicit contribute that students assess 
and reflect about the appropriateness of mates’ ideas (lines 1, 2, 3, 4 and 5). 

Moreover, students are mutually engaged in processes related to task performance (line 6) and student 
takes the lead to start writing a specific section of the text (line 9). 

Furthermore, this student rises group reflection about the main conclusions reached by the group so far 
(line 3) in the wiki negotiation page and s/he launch a group reflection about a key process to follow up with 
finding a novelty solution for the science problem (line 4). 

Table 2. Example of a wiki contribution and the features of L2L2 skills 



Students’ contribution in the wiki negotiation page 

L2L2 skill 

1 

We think that the text written by Ferran and Gerard is really good 

Peer-assess 

2 

and their arguments are convincing and elaborated 

Peer-assess 

3 

All of us think that the heating plant would be a benefit for our community 

Group reflection 

4 

This idea could be used of our initial proposal text? 

Group reflection 

5 

Besides, the text written by Marc and Pepe is also good 

Peer-assess 

6 

because they argue that non-renewal resources pollute more than renewal resources because 

Engagement 

7 

those only use natural resources 


8 

We agree with the previous contributions 

Peer-assess 

9 

At this point what do you think if we start writing the text in the wiki 

Leadership 


However, not all the groups displayed the four L2L2 skills during their collaboration in the wiki and there 
were differences among groups in relation to the presence and proportions of these skills. Besides, we could 
not find a relationship between the presence of features of the L2L2 with the quality and creativity of the 
final collaborative text. 

Therefore, our findings suggest that there is not a direct relation between collaborative creativity 
processes and creative products produced by the group. Although our research objectives focused on the 
promotion of collaborative creativity processes in a wiki environment, we expected a positive incidence of 
these creative processes on the collaborative text written by the group. More detailed qualitative analyses are 
needed to explain these preliminary findings. 
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5. CONCLUSION 

Our findings showed that the wiki environment afforded the development of an effective and creative online 
collaborative learning community. Through the analyses of students’ contributions in the wiki we have 
characterized collaborative creativity processes in science inquiry that includes performance (processes to 
develop a novel way of approaching and understanding the problem) and collaboration (peer collaboration, 
dialogue). In student’s wiki contributions, the four learning to learn together skills took place. The 
collaborative use of the wiki in our study promoted the creation of a collective product in which the users get 
the sense that they were creating a truly shared digital artefact as the product of their collaboration. In doing 
so, the collaborative processes developed by the students encourage them to share their perspectives, to take 
into account other’s opinions, to reflect on other’s opinions and to give a value and an assessment to other’s 
ideas and, subsequently, create new knowledge. 

However, not all the groups displayed the four L2L2 skills during their collaboration in the wiki and there 
were differences among groups in relation to the presence and proportions of these skills. 
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LSQUIZ: A COLLABORATIVE CLASSROOM RESPONSE 
SYSTEM TO SUPPORT ACTIVE LEARNING THROUGH 
UBIQUITOUS COMPUTING 
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ABSTRACT 

The constructivist theory indicates that knowledge is not something finished and complete. However, the individuals 
must construct it through the interaction with the physical and social environment. The Active Learning is a methodology 
designed to support the constructivism through the involvement of students in their learning process, allowing them to 
make analysis, synthesis and evaluations, therefore developing their thinking and reasoning abilities. The technology 
supports active learning through Classroom Response Systems (CRSs), which usually use clickers devices to allow 
students to submit answers to a questionnaire proposed by the teacher. However, these systems have some drawbacks, for 
example not considering the students’ individualities, personal characteristics and needs. In this way, this study describes 
the creation of the LSQuiz, a CRS that implements a collaborative process that allows teachers to propose questions to 
students, who may choose to solve them individually or ask colleagues to help solve them. The LSQuiz applies concepts 
related to ubiquitous computing, such as context awareness and the analysis of the interactions among students to 
determine which student is the most suitable at a given moment to provide help. The system validation indicates wide 
acceptance by students, who consider the possibility of classroom collaboration an important element to support their 
learning process. The experiment indicates that adopting a CRS associated with ubiquitous computing features is a valid 
and effective way to promote active learning. 


KEYWORDS 

Active Learning; Collaborative Learning; Ubiquitous Computing 


1. INTRODUCTION 

There are several proposals and methodologies aimed to make the learning process more interesting and 
motivating for students. One of the most important contemporary pedagogical trends is the constructivism, 
which opposes to the traditional learning in which students passively absorb information. As Becker (1992) 
explains, constructivism has an idea that knowledge is not something finished, also being constituted by the 
interaction of the individual with the physical and social environment. 

A form of practical application of constructivism in the educational environment is the Active Learning 
(Bon well & Eison, 1991; Prince, 2004), characterized by the direct participation of students in their learning 
process (Morable, 2000). As stated by Chickering & Gamsom (1987), in an active learning environment 
students need more than just listen: they need to read, write, discuss, or be actively engaged in solving 
problems. In addition, students must perform analyzes, summaries and reviews to develop their thinking and 
reasoning abilities (Bonwel & Eison, 1991). 

There are some methodologies created to support and promote Active Learning. For example, the 
Collaborative Learning (Collins & O'Brien, 2012), Cooperative Learning (Beck & Chizhik, 2013), Inquiry- 
Based Learning (Schneider, 2012). Problem-Based Learning (PBL) (Vosikanis et al., 2013; Holmgrem, 
2013) and Peer Instruction (Mazur & Watkins, 2013; Zingaro & Porter, 2014). 

It is increasingly common the use of technological devices to implement these techniques. For example, 
the Classroom Response Systems (CRS ) allow feedback from students to the teacher, supporting the PBL and 
the Peer Instruction through the use of clickers, little devices that allow students to submit question's answers 
to the teacher. In these systems, the teacher can receive, analyse and compare a large number of answers, 
giving feedback to all participating students. 
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However, one problem of the CRSs is that they do not consider the students’ individuality and specific 
needs, treating everyone the same way. Characteristics associated with the context, such as location, 
interactions’ history with the teacher and classmates or even the freedom to participate (or not) in a 
collaborative process are not considered. In many aspects, the application of CRSs is closer to the traditional 
educational model than the constructivism, which it should support. Still, considering a technological point of 
view, the clickers have the disadvantage of being limited to a specific technology, not following the rise of 
new technological devices. 

In previous work (Caceffo & Rocha, 2011; Caceffo & Rocha, 2012; Caceffo el al., 2013), we described 
the theoretical propose of a ubiquitous CRS. defined as Ubiquitous Classroom Response System (UCRS). 
The UCRS proposes the use of features associated with ubiquitous computing, such as context awareness and 
multiple device support, like smartphones and tablets, to create a smart collaborative process. This process 
allows teachers to submit questions to students, who may choose to solve them individually or ask colleagues 
to help solve them. In the latter case, the system considers factors such as the students' locations and their 
interaction history (affinity), thus determining which student is the most suitable to be invited to help his 
colleague. The teacher acts in this way as a tutor or mentor, having the ability to view a classroom map, 
intervening if necessary in the collaborative process. 

In this study we present the design, development and practical application of the LSQuiz, a software that 
implements the UCRS. The LSQuiz was developed as a Moodle plugin, supporting any device that has a 
html browser, thus including smartphones, tablets and notebooks. 

Results indicate a wide acceptance for this model application inside the classroom. In addition, the 
combined application of a CRS with ubiquitous computing supports the Active Learning in a more efficient 
and transparent manner. 

This paper is organized as following: section 2 presents the collaborative process definition and the 
LSQuiz features; section 3 presents the LSQuiz implementation; section 5 describes the experimental 
validation of the tool and; in section 6 is presented the conclusions and future works. 


2. LSQUIZ COLLABORATIVE PROCESS 

We defined a collaborative process that supports the interaction among teacher and students through the 
LSQuiz. The 12 steps that compose the collaborative process are: 

1. Teacher creates a questionnaire and submits it to the students; 

2. Students receive the questions on their own mobile devices; 

3. Each student defines his location in a classroom map 

4. Students start to solve the questionnaire; 

5. At any moment, any student (defined as source student) can press a button and access the “help 
request” feature. This feature verifies, based on the location and affinity factor among students, 
which available student (defined as target student) is more suitable to help; 

6. The target student receives a message, indicating the source’s student name, photo, and location. 

7. The target student can accept or not the invitation; 

8. If the target student accepts the invitation, then he is oriented to move next to the source 
student’s place; 

9. Both students solve together the question; 

10. The source student submits, on his device, the question’s answer to the teacher; 

1 1 . Each student, using his device, qualifies the collaborative process as positive or negative. 

12. The target student backs to his original location. Both students continue to answer the 
questionnaire. 

There is more than one possible interaction path from the moment the source student press the “help 
request” button in step 5 until de moment that both students have qualified the collaborative process (step 
11). For example, in step 7 the target student can accept or not the invitation, and in step 11 not necessarily 
both students will qualify the collaborative process at same time. Also, each student will visualize a different 
screen in each one of these steps. A State Machine, as described in section 2.1, organizes these different 
paths, interactions and visualizations. 
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In step 2, each student defines its location through a classroom map. This information supports the step 5, 
where the student’s location is used as a context factor in order to help the LSQuiz to identify which student 
will be invited to help, and in steps 6 and 8, where the invitation is received and accepted (or not) by target 
student. Section 2.2 describes the classroom map feature. 

The step 5 is also supported by the affinity factor, which is an empathy’s measure among students. The 
affinity factor is calculated through the analysis of the proportion of accepted invitations (step 7) and the 
proportion of positive qualifications among students (step 11). The details about the affinity factor 
calculation are described in section 2.3. 

Finally, section 2.4 describes the help request feature. 

2.1 State Machine 


A state machine, with five possible states, manages the LSQuiz collaborative process. The students may have 
two roles in the collaborative process: source and target. The student who asked for help is the source 
student, and the student invited to help is the target student. The LSQuiz adjusts its behaviour according to 
each student's role and state, showing the following elements when necessary: classroom map, showing both 
students location; accept or reject help buttons, and qualification buttons (positive and negative) for the 
collaborative process. The latter two use as a template the Facebook’s Like button to indicate the acceptance 
or positive qualification and a Dislike (upside down Like image) to indicate the help rejection and negative 
qualification. 

Figure 1 shows the state diagram, describing the relations and transitions among states and the respective 
elements displayed regarding each student's states and roles. 


Start 


End 



Figure 1. LSQuiz diagram states 

The starting state is triggered when, in the step 5 of the collaborative process, the source student presses 
the help request button. The State 0 corresponds to when the target student receives an invitation to help the 
source student, as described in the step 6 of the collaborative process. This invitation contains a classroom 
map with the source student’s location, name and photo. If the target student refuses to help, the collaborative 
process ends (end state). Moreover, if the target student accepts the invitation (collaborative process step 8), 
then the state machine moves to State 1 . In this state the target student receives a message containing the 
source student’s location, indicating that he should immediately go there to help the source student. Also, to 
both students are displayed qualifications buttons. 

The states 3 and 4 relate to the situation when only one of the students have qualified the collaborative 
process. In this case, the student that have made the qualification is considered in the End State, being free to 
collaborate with other students as source or target student. However, the student that have not yet qualified is 
considered busy, not being able to make help requests or to be invited to help other students. 

The State 5 corresponds to the situation when both students have qualified the collaborative process and 
can continue answering the questionnaire, as indicated in step 12 of the collaborative process. 
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2.2 Classroom Map Feature 

In the step 3 of the collaborative process, the system asks the students to report their approximate classroom 
location. The procedure is to move a box containing his or her name above the approximate location as 
shown in Figure 2a. The teacher can, at any time, view each student location trough a classroom full map, 
which shows the students names and their photos placed above their locations, as shown in Figure 2b. 



Figure 2. Students indicate their classroom location (Figure 2a) and; teacher’s classroom map view (Figure 2b). 

Still, the help request step (step 5) of the collaborative process uses the classroom map to indicate to the 
target student the source student’s location. The student location feature also provides the information related 
to the distance between each one of the students. The LSQuiz uses these data to define which student should 
be invited to help the source student, as explained in section 2.4. 

2.3 Affinity Factor 

The Affinity Factor (AF) defines objectively the affinity degree (empathy) among students during the 
collaborative process. The AF, in conjunction with the students’ location, determines which students should 
be invited to help, as described in section 2.4. The priority is to assign students with higher affinity to work 
together when possible. 

All students have an AF related to others, ranging in scale from 0 (lowest affinity) to 1 (highest affinity). 
At the beginning of each collaborative process, the Formula 1 is applied, updating the AF according to some 
context factors, like the proportion of accepted help invitations and the positive qualifications among 
students. Formula 1 uses a similar approach as proposed by Levis et al. (2008). 

Formula 1. AF = 0.3HR + 0.7PQ 

Where: 

• HR = Proportion of help requests accepted 

• PQ = Proportion of positive qualifications 

In Formula 1, HR indicates the percentage of help requests accepted related to the total help requests 
among these students, and PQ indicates the percentage of positive qualifications related to the total of 
collaborative process qualifications among them.The weights are 30% for HR and 70% to PQ. PQ weight is 
higher in order to focus the collaborative process qualification, since the help request acceptance can be 
associated with factors not necessarily related to the students’ affinity ( e.g target student is busy with 
another task at the request time). 

2.4 Help Request 

At any time, any student that has difficulties in solving the questionnaire can press a button to request 
assistance from another student (step 5 of the collaborative process). After pressing the button, the LSQuiz 
automatically defines which student is the most suitable to be invited to help the source student through the 
following factors: students that are available, i.e, are not helping or being helped by anyone; students 
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distances from the source student (defined by the classroom map functionality, as indicated in section 3.1) 
and the AF among students. 

If there are students available to help, LSQuiz sorts the students according to their distance from the 
requesting (source) student. From the student with the shortest distance, the LSQuiz identifies the first 
student whose affinity is greater than the average affinity value calculated among all students available and 
the source student. Therefore, this student is invited to help the source student. 

The target student receives an invitation message, displaying the classroom map with the source student 
location and identification, as showed by Figure 3: 

Turin wants your help to solve a 
problem. Can you help him? 

■3 P 



Figure 3. Invitation message displayed to target student. 

If the target student not accepted the invitation, LSQuiz sends a message to the source student stating that 
his colleague is not available for help. However, if the target student accepted the invitation, the source 
student receives a message stating that he must wait for the target student help (see Figure 4a). In addition, 
the target student receives a message asking him to move to the source student location (see Figure 4b). For 
both students are also displayed qualification buttons for the collaborative process evaluation. 



Figure 4. Invitation acceptance and collaborative process qualification steps 


The data gathered in this step, related to the invitation acceptance and collaborative process qualification, 
are used to support the AF calculation, as described in section 2.3. 


3. LSQUIZ IMPLEMENTATION 

The LSQuiz was implemented as a Moodle 1 plugin. One difficulty encountered during the collaborative 
process implementation was the need to do a refresh in a specific screen region, thus keeping unchanged the 
rest of the screen contents. Thus, we adopted an approach based on AJAX technology, which allows the 
client to submit requests to the server and display the response data on specific and pre-determined screen 
area. 

As described in the next section, this solution proved satisfactory to the LSQuiz, an academic prototype 
with relatively few users. In large-scale application systems, however, solutions that are more robust, like the 
Long Polling or WebSocket (Idol, 2013) should be considered. 


1 Moodle is an open-source LMS - Learning Management System. Available at: https://moodle.org/ 
Accessed: June 2014 


67 


ISBN: 978-989-8533-23-4 © 2014 IADIS 


4. VALIDATION EXPERIMENT 

4.1 Methodology 

In order to validate the LSQuiz application in real learning environment, an experiment was conducted with a 
group of college students (discipline laboratory of public policies). A small class, with 10 students was 
selected making more effective the initial application procedure and prototype analysis. 

The methodology of this experiment followed the recommendation proposed by Rocha & Baranauskas 
(2003), being implemented in the following stages: preparation, introduction, testing, and final session. The 
experiment methodology is as follows: 

• Preparation: In previous classes prior to the experiment, the teacher identified that most students 
bring to class some mobile device (e.g., smartphone, notebook or tablet) with internet access. Just in 
case, some devices were provided for ensuring back up if necessary. In addition, in order to prepare 
the experiment next steps, the students were registered in the Moodle system, including their profile 
photo. 

• Introduction: After the initial presentations, the students were asked to access the Moodle to answer 
a questionnaire, composed by 5 questions related to the current discipline topic. 

• Testing: In the questionnaire first question, students were oriented to locate themselves in the 

classroom map. Then they had the freedom to take the questionnaire at their pace, requesting help 
from colleagues if necessary. At any time, the teacher can access the classroom map and visualize 
all students’ locations. 

• Final Session: At this step, students and teacher received an evaluation form. The qualitative and 
quantitative data obtained from this form is discussed in the follow session 5.2. 

4.2 Results 

In general, the results show the collaborative model proposed by LSQuiz was widely accepted. Figure 5 
indicates that students agreed on the following statements: "It was simple and practical inform the system my 
location in the classroom" (Figure 5a); and "The classroom map presented a correct representation of my 
position and my colleagues in the classroom." (Figure 5b). 



Strongly Partially Indifferent Partially Strongly 

Disagree Disagree Agree Agree 


Figure 5a 



Strongly Partially Indifferent Partially Strongly 
Disagree Disagree Agree Agree 


Figure 5b 


Figure 5. Students’ opinion about the classroom map feature 


However, the qualitative data analysis identified an issue related to the step where students should locate 
themselves in the LSQuiz. As one student stated: “The classroom arrangement and number of seats and rows 
was different in LSQuiz.”. The problem is related to the fact that students tried to locate themselves 
accurately in the LSQuiz classroom map, which was not the original software objective. Thus, in future work 
is intended to decrease the classroom map accuracy, making it clear to students that they should inform the 
LSQuiz their approximate classroom location. 

Regarding the collaborative process, qualitative analysis indicated that its use took place without major 
problems. Figure 6 indicates that students agreed on the following statements: "The move process to the 
colleague location was quiet." (Figure 6a); and “The collaborative process qualification, through the 'Like' 
and ‘Dislike’ buttons, was simple and clear to be held "(Figure 6b). 
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Figure 6. Students’ evaluation the help (Figure 7a) and qualification (Figure 7b) procedure 


Still, all of the respondents stated to be positive the opportunity to request assistance to colleagues to help 
solve an activity. However, the qualitative data analysis identified a situation where a particular student, after 
refusing to help his colleague, was again selected by the LSQuiz to help the same student. 

We identified that this is related to the LSQuiz algorithm, which in the AF calculation (see section 2.3) 
considers most important collaborative process qualification than the acceptance or not of the help request. In 
order to minimize this problem, LSQuiz future versions will consider: a) a clause that inhibits a student who 
did not accept help a colleague to be chosen again, in the same class, to be invited to help the same student 
and b) a change in the AF calculation, leaving both factors (acceptance and qualification percentages) with 
the same weight of 50%. 


5. CONCLUSION AND FUTURE WORK 

The LSQuiz implementation and evaluation indicate the importance of the model focused on students' 
autonomy, where students can interact and collaborate with their peers according to their will. The 
collaboration is supported by context factors, like the students’s location and affinity. This approach aims to 
support the student’s individuality and specific needs, as each student can at any moment ask for help or 
accept or not accept an invitation. 

Technology is a crucial factor to support this approach since the automation of the collaborative process 
would not be possible without the use of mobile devices. Regarding the experiment, teacher and students 
approved the use of the LSQuiz prototype, stating its effectiveness in the teaching and learning process. 

According to students, the classroom has become more dynamic and interesting through the 
communication and collaboration process proposed by the LSQuiz while the teacher related an increased 
participation and motivation of students. These reports support the main LSQuiz goal, which is to create an 
Active Learning environment where students have an active participation in their learning process. 

Teacher and students also related as a project risk factor the classroom size and the number of students. 
According to them, the experiment happened in a small room with few students, thus favoring the mobility 
and communication among them. They concluded that, in other condition, the LSQuiz application could not 
be so positive. Thus, based on user’s opinions and following the spiral prototyping methodology (Rocha & 
Baranauskas, 2003), we are planning the next experiment in a large class, involving more students. 

Concerning the classroom map, students related some difficulty in identifying their exact location in the 
system. Ideally, following ubiquitous computing concepts, the determination of student's location should be 
an automated process, without the user’s interference. However, this approach is not possible today due the 
technological constraints related to the current mobile devices. LSQuiz future versions will have a classroom 
map with lower accuracy, making clear that students should use their approximate classroom location. 

Also, the experiment related on this paper describes the outcomes from just a single session. The results 
conceptually validated the LSQuiz proposal the application. In future studies we intend to apply this system 
for one semester, which will allow us to analyse the impacts of this methodology in the teaching and learning 
environment for a longer period. 

Still, future work involves the students’ classification according to their learning styles, following the 
Felder & Silverman model ( 1988). We expect that providing customized content for each student, according 
to his preferred learning style, is an effective way to individualize the learning process, thus increasing 
motivation and promoting the active learning in a more effective way. 
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ABSTRACT 

The very nature of wiki-based collaborative writing around a topic -related content is the interaction of wiki functionality, 
content, and collaborative learning, and how the functionality is used to create the content collaboratively. This way of 
looking at wikis provides a theoretical model that helps researchers and educators to identify a number of category 
actions that can be carried out on wikis when students perform collaborative writing activities. This work aims at using 
the proposed model to evaluate the impacts of MediaWiki on collaborative writing among students in teacher education. 
Pedagogical implications are drawn from the results and future research actions are envisaged to enhance the quality of 
wiki-based collaborative writing in teacher education. 


KEYWORDS 

Collaboration, collaborative learning, collaborative writing, category action, MediaWiki, wiki 


1. INTRODUCTION 

Wiki was invented by Ward Cunningham in 1994 (Leuf, & Cunningham, 2001). It is as a type of social 
software that offers new opportunities to create and edit web page content collaboratively. Wikis have been 
promoted as collaborative writing tools in educational settings and society at large. In fact, wiki technology 
matches the evolution of the networked society that requires team work, collaborative learning, and 
collaborative writing skills. Universities and educational institutions have therefore the responsibility to 
engage students in collaborative learning and writing. A large body of research exits on wikis and their effect 
on collaborative writing. In teacher education, the research literature reports on a number of wiki -based 
collaborative writing applications, such as online coursework, teacher evaluation, class project, joint article, 
digital stories, and many other application examples (Austin et al, 2010; Deters, Cuthrell, & Stapleton, 2010; 
Chao, & Lo, 2011; Every, Garcia, & Young, 2010; Grant, & Mims, 2009; Mindel, & Verma, 2006; Parker, & 
Chao, 2011; Witney, & Smallbone, 2009). However, the claim that wikis support collaborative writing has 
not yet been evidently confirmed in educational settings, despite studies reporting on successful experiences 
(Karasavvidis, 2010). The main goal of this work is to use a theoretical model to analyse the impacts of wikis 
on students’ collaborative writing in teacher education. The model results from the interactions of wiki 
functionality, wiki content, and collaborative learning, and includes a taxonomy of category actions that can 
be carried out on wikis. The main data collection and analysis method is the wiki history log. In addition, the 
work uses a survey questionnaire and comments posted on the discussion page as supplementary methods. 
Pedagogical implications are drawn from the results. Research actions for the near future are also envisaged 
to enhance collaborative writing in teacher education. 


2. THEORETICAL MODEL 

Wiki-based collaborative writing emerges from the interactions of wiki functionality, wiki content, and 
collaborative learning (Hadjerrouit, 2013a). Wiki functionality is used to structure the content of the wiki by 
means of a simplified HTML language (Lamb, 2004; Tetard. Patokorpi, & Packalen, 2009). It includes a 
discussion page for written communication and reflections, and a history log that tracks all students’ actions 
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carried out on the wiki. Among a plethora of wiki tools, MediaWiki was chosen as a platform for 
collaborative writing as it incorporates all features mentioned above. MediaWiki is restricted to university 
members, making it appropriate for education. Wiki functionality enables students to create a wiki that can be 
evaluated using usability criteria. Collaborative learning, which describes a process generated by student 
groups, enables the collective creation of wiki content. The content dimension is important for many reasons. 
It may stimulate students to be engaged in collaborative learning. It may also motivate them to critically 
discuss and use wiki functionalities to create and edit the content collaboratively. Finally, it may challenge 
students to take into account the characteristics of the target users. Figure 1 shows the interactions of wiki 
functionality, wiki content, and collaborative learning. 



Figure 1. Collaborative writing results from interactions of content, functionality, and collaborative learning 

Students carry out actions on the wiki when they perform collaborative writing. Figure 2 shows a 
taxonomy with originally 13 category of actions, of which the following 10 are used to analyse collaborative 
writing activities (Pfeil, Zaphiris, and Ang, 2006, pp. 101). 



Figure 2. Taxonomy of category actions 


Genuine collaborative writing is a matter of changing, clarifying and reworking each other’s contributions 
to the wiki (Hadjerrouit, 2013b). In contrast, work where participants divide the wiki task in subtasks among 
themselves, and develop the content independently of each other without changing the content produced by 
peers, cannot be considered as genuine collaboration (Ibid). The wiki history log provides in-depth 
information about the types of actions that the students carry out on the wikis. The log is an instrument that 
helps instructors to assess students’ collaborative writing activities in terms of work distribution, type of 
actions across the categories of the taxonomy described above, and timing of contribution as well. 


72 





1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


3. RESEARCH QUESTIONS 

This work aims at evaluating the impacts of MediaWiki on students’ collaborative writing in teacher 
education. On the basis of the results, pedagogical implications and future research actions are suggested. 
The work addresses three research questions: 

1) What are the impacts of MediaWiki on teacher students’ collaborative writing activities? To 
investigate this question, it is necessary to account for three sub-questions: 

• What is the level of work distribution among members of each student group? 

• What are the categories of actions carried out on the wikis? 

• What is the timing of contribution of each student group? 

In addition to the level of collaborative writing, the work aims at answering two supplementary questions: 

2) Which pedagogical implications can be drawn from the results to foster collaborative writing by 
means of wikis? 

3) Which research actions need to be taken to enhance wiki-based collaborative writing? 


4. METHODS 

Data came from 16 students enrolled in a course on Web 2.0 technologies in teacher education in 2012. The 
participants were divided into 6 groups of 2 to 4 students. None of the students had experience with wikis or 
collaborative writing. The wiki tasks included subjects such as geography, history, mathematics, and science. 
The main data collection method was the history log that tracks all students’ actions carried out on the wikis. 
Data found in the log was analysed in terms of the category actions described in the theoretical model, that is 
add and delete content; add, fix, and delete links; format, correct grammar and spelling mistakes, and clarify 
content. In addition, two supplementary methods were used: a survey questionnaire with open-ended 
questions, and comments posted on the discussion page of the wikis. The questionnaire was used to obtain 
information on technical usability of wikis, motivation to use wiki, level of collaboration, and quality of wiki 
content. To assess their responses, the survey used a five-point Likert scale to measure the extent to which 
they strongly agree, agree, neither agree or disagree, disagree, or strongly disagree. Finally, the content of the 
discussion logs were analysed in terms of number and quality of comments provided. 


5. RESULTS 

5.1 Level of Work Distribution 

Table 1 presents the level of work distribution made by each student (S_l, S_2, S_3, S_4) in the respective 
groups (G_l, G_2, G_3, G_4, G_5, G_6) in terms of number of actions carried out on the wikis. 


Table 1. Distribution of work among members of each group in terms of number of actions 
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The table shows that the workload of the groups was not equally distributed between all members. In 
groups of 2 students (G_4, G_5 and G_6), the results indicate that one member of the groups assumed nearly 
68-61% of the workload, while the other students contributed to 38-32%. In groups of 3 students (G_2 and 
G_3), two members (S_l and S_2) assumed more than 85% of the workload, while S_3 contributed to only 
11-14%. Finally, in the group of 4 students (G_l), the results show that 3 students (S_l, S_2, and S_3) did 
most of the work (86%), while S_4 contributed only to 14 %. The results reveal that there is a huge variation 
in terms of number of actions among the best students in the groups ranging from 640 (S_l in G_4) to 80 

(S 1 in G_l). The analysis of work distribution does not reveal the extent to which students collaborated to 

create the wikis. A categorization of actions is therefore necessary to analyze the level of collaboration. 

5.2 Category of Actions Carried out on the Wikis 

Figure 3 shows the number and percentage of actions that fell under each of the 10 categories investigated. 
The table reveals that the most common category of action was format (668 actions, 23.69% of all actions), 
followed by addition of content (589, 20.62%), addition of links (505, 17.68%), clarification of content (334, 
12.04%), fixing of links (221, 7.73%), deletion of content (207, 7.25%), style/typography (134, 4.69%), 
deletion of links (77, 2.70%), spelling (68, 2.39%), and finally grammar being the last common category 
actions (43, 1.51%). Note that a single edit may involve several actions, such as add content, delete a link, 
clarify content, or correct spelling mistakes. 



Format: 23.63% 
A<i«l_coiiteiit: 20.62% 
Add_link: 17.68% 

Clarify content 12.04% 
Fix_link: 7.73% 

Delete content: 7.25% 
Style Typography: 4.63% 
Delete link: 2.70% 
Spelling: 2:33% 
Grammar: 1.51% 


Figure 3. Category of actions in terms of number of actions and percentage of total actions 


5.3 Timing of Contribution 

Figure 4 shows the timing of contribution of each group in terms of number of actions per week. The results 
show that all groups worked much as the deadline for submitting their work approached (March, 15), except 
for group 6. This is reflected in the average number of actions per week carried out on the wikis. 
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5.4 Summary of Results 

A careful analysis of the history log in terms of the category actions described in the theoretical model 
reveals that there is no evidence that the students truly collaborated. Instead, format, style/typography, 
add/delete content and links were the most actions that were carried out on the wikis. Furthermore, the 
workload was not evenly distributed among all members of the groups. The timing of contribution also 
indicates that all groups worked much as the last day for submitting their wiki approached. Clearly, 
collaborative writing was performed on a superficial level by formatting the presentation of the wikis, adding 
content and links. In contrast, genuine collaborative writing by reworking each other’s contributions was not 
a frequent activity. As a result, individual work on the wikis was more evident than collaboration among 
teacher students. 

The survey questionnaire shows that the students were globally satisfied with their wikis and the 
potentialities of MediaWiki to foster collaborative writing. However, responses to open-ended questions 
revealed a number of problems that need to be addressed. First of all, MediaWiki lacks a user-friendly editor 
that may facilitate collaborative writing. It also lacks advanced functionality that supports the development of 
more usable wikis. Secondly, most students think that the discussion page is not good enough to promote 
genuine communication and critical reflections. As a result, students still value face-to-face dialogue, and 
discussions by means of other communication channels. The low level of collaborative writing revealed by 
open-ended questions was globally confirmed by the issues addressed in the discussion page. The results 
achieved so far are in line with similar research work (Cole, 2009; Grant, 2009; Elgort, Smith, & Toland. 
2008; Hadjerrouit 2012; Hadjerrouit 2013b; Judd. Kennedy, & Cropper, 2010; Karasavvidis, 2010; 
Neumann, & Hood, 2009). These studies point out to several hypotheses in an attempt to explain the low 
level of collaboration when using wikis in educational settings: unfamiliarity with wikis, lack of experience, 
dominant learning paradigm, limited student contribution, reluctance and resistance to use wiki, lack of 
motivation and engagement, time management, problem of ownership, and lack of an appropriate pedagogy. 


6. DISCUSSION 

Research question 1 addressed the impacts of MediaWiki on collaborative writing in terms of work 
distribution, actions on the wikis, and timing of contribution. The results show that wikis alone cannot make 
collaborative writing automatically happen, unless a collaborative learning approach to wikis is adopted in 
educational settings. Based on the results, research question 2 and 3 focus on pedagogical implications and 
future research actions to help teacher students benefit from using wikis for collaborative writing. These are 
addressed in the following sections. 

6.1 Pedagogical Implications 

The benefits of collaborative writing are underpinned by collaborative learning and associated Vygotsky’s 
socio-cultural theory that assumes that learning occurs through collaboration and information sharing in 
authentic contexts (Vygotsky, 1978). Collaborative learning is basically considered as superior to individual 
learning (Witney, & Smallbone, 2011). As such wikis enable students to share information, collaborate, and 
communicate with each other. Wikis may stimulate students to work together to create a collective document, 
a joint project, essay or article. However, collaborative writing cannot develop fully, unless collaborative 
learning approaches are adopted in educational settings. Those approaches to wikis will increase the 
likelihood of students’ engagement and participation in wiki-based collaborative writing. Nevertheless, 
collaborative writing will not work successfully unless students are accustomed to collaborative practices. 
Therefore, wikis should be developed in collaboration to realize shared knowledge, and not be limited to each 
individual student (Grant, 2009). To achieve this, students should be given more time and training 
opportunities to familiarize themselves with those practices. 

As collaboration and critical reflections are necessary conditions for using wikis, students should discuss 
the writing tasks in terms of usefulness of content, relevance of literature, and adaptation of the wikis to the 
users’ needs. Clearly, students should firmly grasp their knowledge in order to make their wikis more 
attractive for the target users. Furthermore, they should focus less on selecting content from Wikipedia and 
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other Web sites than critically discussing their ideas. However, some language proficiency is required to 
improve the quality of collaborative writing and make the writing process easier, especially for students with 
technical background (Li, & Zhu, 2011). 

Another implication is that assessment plays an important role in evaluating students’ contributions to the 
wiki. Both quantitative and qualitative assessment may be used. Assessment forms may be self-assessment 
or/and peer-assessment on an individual basis or in groups. Also the close integration of the wikis and 
assessment goals, whether and how the wikis will be assessed individually or in groups, and which 
assessment forms are used, may motivate students to effectively engage in meaningful collaborative writing 
(Hadjerrouit, 2013b). In this regard, motivation is an essential component of collaborative writing, and must 
be seen in relation to the wiki content itself, whether it is intrinsically interesting, highly relevant and 
meaningful to the students. 

Finally, wikis as a type of social software can stimulate students to express their ideas better than using 
another technology. However, students need to be aware of the added value of wikis in terms of collaborative 
capabilities compared to other communication systems such as LMS. They also need to critically judge the 
limitations of wikis when communicating with each other. As the wiki discussion page in its present form is 
not the ideal arena through which to reflect ideas, students still need face-to-face meetings and oral 
communication by means of other channels, e.g., mobile phone and email, or in conjunction with similar 
ways of collaborating, including social software such as Facebook and Blogs. As a result, a pedagogical 
model that combines wiki technology, face-face interactions, and social software may provide the most 
beneficial communication scenario for teacher students. Nevertheless, current wiki platforms should be 
improved to include advanced functionality that enables students to express their ideas in collaboration with 
peers. Clearly, the limitations of wiki technology should not disturb the flow of discussion and 
communication threads. 

6.2 Future Research Actions 

Many research actions can be envisaged in the near future to foster wiki-based collaborative writing in 
education. Firstly, wiki technology needs to be further developed to ensure a smooth interaction with the 
students in order to free more resources for collaborative writing. Usability aspects are still an important 
factor for evaluating the value of wiki tools thereby confirming the continued validity of previous research 
studies (Desilets, Paquet, & Vinson, 2005; Kickmeier-Rust, Ebner, & Holzinger, 2006). Despite some 
progress in designing the wiki user interface to assist non-technical users, work remains to be done to 
improve the usability of wikis in terms of user-friendly editors, structure of content, navigation, or visual 
presentation format. Clearly, technical problems should not constitute a source of frustration for students 
(Jones, 2010). A smooth interaction with the wiki tool requires not only a user friendly editor, but also 
extended functionality and advanced features, e.g., communication and management capabilities. Likewise, 
future wiki tools in education should be designed to support teaching and learning processes, and not only 
commercial purposes (Wichmann, & Rummel, 2013). 

Another research action is the development of a wiki-based pedagogy to ensure a better use of wikis in 
education (Cole, 2013; Hadjerrouit, 2013b). The question, which faces researchers and educators today, is 
not why or whether to use wikis, but when and how, that is issues of wiki pedagogy (Walsh, 2010). In this 
regard, the importance of a socio-cultural learning approach to wikis based on Vygotsky’s ideas cannot be 
underestimated, since putting students together does not automatically result in collaboration. Extending wiki 
to include a socio-cultural approach to learning requires the integration of wikis into a pedagogical strategy 
that supports genuine collaborative learning. A pedagogy that suits wikis can engage students in collaborative 
work and group dynamics to a greater benefit for all members of the groups. This approach to wikis requires 
a thoughtful interweaving of content, technology, and collaborative learning as described in the theoretical 
model to the benefit of collaborative writing. In a wiki-based learning environment, students have to actively 
engage in design practices and inquiry as they develop their wiki. In this environment, the role of the teacher 
is to create an atmosphere of confidence that stimulates students to collaborate for the benefit of the groups, 
as well as provide specific guidance to assist them in the writing process. Ultimately, the process of creating 
a wiki of high quality needs to be carefully planned by teachers to sustain, manage, and motivate student 
participation in collaborative writing activities (Allwardt, 2011; Huang, &Nakazawa, 2010). 
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Finally, following the socio-cultural approach to wikis, a near-future research action is to help students 
become more familiar with wikis and more skilled in collaborative learning and writing (Harsell, 2010). 
Indeed, a critical factor of success is the students’ preparation for collaborative writing, and familiarization 
activities with wikis (Minocha, & Thomas, 2007). In the networked society that is grounded on team work, 
training in collaborative learning and acquisition of collaborative skills cannot be restricted to wikis alone, 
but will be possible using appropriate methods, such as allow students with different backgrounds discuss 
and integrate different aspects of a topic-related content or develop mutual understanding of complex issues 
of the topic, and thereby add to each other’s knowledge. Another way to foster collaborative learning may be 
the discussion of students’ summaries through the study of wiki-related topics (Tetard, Patokorpi, & 
Packalen, 2009). Group-based tasks and similar work may provide opportunities for wiki-based collaborative 
learning and writing. But still, it is likely that students will need constructive feedback and guidance in seeing 
collaborative learning as a part of their work (Wake, & Modla, 2012). 


7. CONCLUSION 

Wiki technology alone is simply not enough to foster collaborative writing in teacher education. Rather, 
wiki-based collaborative writing requires a thoughtful understanding of the complex relationships between 
wiki functionality, wiki content, and collaborative learning. Based on the results, pedagogical implications 
and possible research actions for the near future are suggested to successfully foster wiki-based collaborative 
writing in educational settings. The work will evolve in many directions to further investigate wiki -based 
collaborative writing. In terms of aims, the work will be extended to include various topics in higher 
education. In terms of scope, several case studies will be undertaken in the future to explore wiki-based 
collaborative writing in more details and depth. In addition, case studies with a larger population of students 
will be considered in future research to ensure more reliability and validity of the results. In terms of 
theoretical perspective, the model for evaluating the value of wikis for collaborative writing will be refined to 
strengthen the relationships between wiki content, wiki functionality, and collaborative learning. The 
interweaving of these three components can create a new form of knowledge that students can bring into play 
any time they produce wikis for teaching and learning purposes. In terms of methods, the instruments being 
used, that is survey questionnaire and discussion comments, and the taxonomy of category actions, will be 
refined by considering more specific questions and issues associated with collaborative writing. Other 
methods such as focus groups and interviews may be used to support data collection and analysis. 
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ABSTRACT 

This paper presents the results of a study aimed at investigating how and why secondary school students use spelling and 
grammar checkers to aid them in their English writing. The study was a result of close observation over many years of 
how some students use computers effectively to support their writing, while others make just as much use of computers, 
but struggle to read and write in English. Six Year 8 students (as case studies) were selected to participate in the study 
based on their performances in validated standardised literacy tests. Two case studies in each of the scoring categories of 
‘high’, ‘medium’ and ‘low’ were observed during their English writing activities with and without the use of computers. 
The students were interviewed about their self-ratings and experiences. Their teachers were also surveyed and 
interviewed to validate points of concerns. The students’ perceptions indicated that the regular uses of computer tools 
(spelling and grammar checkers) at Years 8 and 9 aided them in their English writing, but they did not retain the skills 
that the tools offered. 


KEYWORDS 

Students, reading, writing, computer literacy, spelling and grammar checkers. 


1. INTRODUCTION 

Since computers became part of domestic, industrial and educational environments, new and innovative 
technologies have created changes and challenges in all fields. Children (the Net-Generation) are more 
perceptive and willing to exploit the electronic devices by engaging in the digital culture to construct self- 
identity and community (Mountifield: 2006). The school aged children live in the emerging world (Atkin 
1998), a world of transition and with a global and local focus. The use of electronic medium/ICT has 
influenced the way young people perceive and transform the traditional written language into a language of 
their own, shaped by short conventional text messaging and online -chatting (Harris 2008). The Net- 
Generation has the opportunity to use the computer and their tools (spelling and grammar checkers) to 
present their work in a traditional form rather than in the variety that they invented. Many students trust the 
spelling and grammar checkers because they are very unsure of their own spelling and grammar (Sinclair 
2010 ). 


2. THE PURPOSE OF THE STUDY 

The purpose of the study was to investigate how selected secondary school students’ use of computer-based 
spelling and grammar checkers relates to their language literacy development. In relation to language, 
literacy is commonly defined as the ability to read and write effectively in a range of contexts (Forster 2009: 
12). In today’s society, both language and computer literacies are of major concern to scholars and educators. 
The study explores the relationship between the development of students’ language literacy skills and their 
computer use in both school and domestic environments, since school tasks and activities are digitally 
produced with the support of resources from the internet. 
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3. RESEARCH METHODS 

I have adopted a triangulation approach including both quantitative and qualitative methods using a mixed 
methods design for this two-phase study. In Phase 1, the selection of the six case studies was conducted 
through administering standardised literacy tests prepared by the Australian Council for Educational 
Research (ACER) to three Year 8 classes of 65 students. The tests provided information relating to students 
performance of their reading and writing and a survey sought their perceptions of the usefulness of the tools. 
The quantitative data from the first phase was sequentially integrated into the second phase where qualitative 
methodology was used to obtain a deeper understanding (McMillan and Schumaker 2006, Creswell and 
Garrett 2008) of the influences of the tools on the students’ English writing. To elicit qualitative data, I 
observed the students in their classes, analysed their writing and interviewed them and their teachers. 


4. RESULTS 

The six students who made up the case studies were three girls and three boys. They were selected and 
categorised into three levels according to their ACER test results. A boy and a girl from any given 
performance level were selected from the same class. The case studies are referred to by a code that 
represents gender and test performance: G for girl, B for boy; H for High, M for Medium and L for Low test 
performance. Their teachers are referred to by a code TH (Teacher of the High achievers, TM (Teacher of the 
Medium achievers) and TL (Teacher of the Low achievers). Their identities are coded as shown in Table 1. 


Table 1 . Case study participants. 


Students 

Gender 

Test performance group 

Case Study Code 

Teacher Code 

1 

Girl 

High 

GH 

TH 

2 

Boy 

High 

BH 

TH 

3 

Girl 

Medium 

GM 

TM 

4 

Boy 

Medium 

BM 

TM 

5 

Girl 

Low 

GL 

TL 

6 

Boy 

Low 

BL 

TL 


The remainder of this paper is divided into three parts. The first part presents the profiles of the six case 
studies in terms of their achievements on the ACER tests. The second part focuses on the teachers’ views of 
the case studies’ English reading and writing skills. The third part reveals the strategies that the case studies 
employed while they composed their English writing. In Phase 2, the in-depth individual interviews were 
conducted and recorded after close observation of the participants in the classrooms. 

4.1 Part 1 - The Case Studies’ ACER Tests Results 


Table 2. ACER test results for the case studies. 


Case studies’ ACER tests results 

Case studies 

Vocabulary 
Max = 40 

Vocabulary 

Percentile 

Adjusted 
Comprehension 
Max = 43 (Raw 
Comprehension 
Max = 21) 

Comprehension 

Percentile 

Writing 
Max = 30 

GH 

32 

98 

30 

89 

26 

BH 

31 

97 

34 

93 

27 

GM 

20 

66 

20 

53 

20 
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BM 

18 

57 

21 

57 

16 

GL 

12 

26 

15 

35 

12 

BL 

10 

17 

16 

38 

6 


The results in Table 2 reveal a pattern differentiating between the performances of the case studies on the 
tests. The percentile ranking of the reading vocabulary results was calculated using the guidelines supplied by 
the test producers (ACER). GH’s and BH’s vocabulary results were ranked at the 97 th - 98 th percentiles. As a 
result they were labelled the high achievers, while, GM’s and BM’s reading vocabulary results were ranked 
at the 57 th - 66 th percentiles and they were labelled as the medium achievers while GL and BL were labelled 
as the low achievers as their results were ranked at the 17 th - 26 th percentiles. A similar pattern characterised, 
the grouping of the case studies’ results in the writing test, as can also be seen in Table 2 above. 

No percentile was calculated for the writing test. The writing pieces were assessed on the content/context, 
language and “on balance” total (the scores for the content, language and on balance are added). As shown in 
Table 2, GH’s and BH’s writing pieces met most of criteria in the ACER guidelines. They achieved scores 
between 26 and 27 out of 30 in the writing test and were classified as high. The writing by GM and BM met 
some of the criteria and achieved scores between 16 and 20 out of 30 and were classified as the medium. GL 
and BL achieved between 6 and 12 out of 30 on their writing pieces and were classified as low. The high 
achievers wrote lengthy and expressive pieces of writing on each of the topics. The medium achievers wrote 
less but demonstrated a distinguishable story-line and attempted to incorporate the characters’ names but the 
characters were not well defined and showed little evidence of selection and control of the content to achieve 
specific purposes. The low achievers’ writing was very brief and did not contain many different ideas. Their 
writing consisted of a few sentences showing basic understanding of the demands of the task. They also had 
difficulty with language skills such as vocabulary, spelling, grammar and handwriting. 


4.2 Part 2 - The Case Studies’ Ratings of their Reading and Writing and their 
Teachers’ Perceptions 

Concurrent with students filling in the questionnaire, their teachers provided evaluations of their students’ 
reading and writing capabilities. The case studies’ responses to the survey are compared to the perceptions of 
their teachers in Table 3. 

Table 3. Case studies’ self-rating in reading and writing in English and their teachers’ perceptions. 


Case Studies’ self-ratings in reading and writing in English and their teachers’ perceptions 


Case Studies’ self-rating 


Teachers’ perceptions 

Case 

studies 

Self-rating of 
reading in 

English 

Self- 
rating of 
writing in 
English 

Self-assessment of 

vocabulary 

development since 

Grade 3 

Teachers 

What assumptions can you 
draw from your experience 
as a classroom teacher about 
the students’ reading? 

GH 

Very good 

Very good 

A great deal. The wide 
reading I engaged in was of 
a great help. 

TH 

She reads widely and has high 
expectations of herself. 

BH 

Very good 

Very good 

I had to work hard on my 
vocabulary. My parents are 
the key as well as my 
reading of books. 

He is a big challenge - he reads 
widely and is knowledgeable 

GM 

Very good 

Good 

My vocabulary has not 
developed as much as I 
would like it to. 

TM 

She is OK but is capable of 
reading better than she has been. 

BM 

Very good 

Good 

Not a great deal. 

He can read better if he 
consistently practises reading. 

GL 

Good 

Poor 

Definitely not. 


She is no challenge, does not read, 
can be frustrating. She is very 
quiet. 
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BL 

Poor 

Poor 

Definitely not. 

TL 

He is a big challenge not only in 






the lack of reading but also 






keeping the same exercise book 






from one lesson to the next. 


Table 3 displays the students’ self-rating in English (reading and writing) and their teachers’ perceptions 
of their literacy capabilities. High achievers rated both their reading and writing in English as very good. The 
medium achievers rated themselves from very good to good while the low achievers rated themselves from 
good to poor. They were also asked to comment on their vocabulary development. The comparisons are 
almost the same between the case studies’ and their teachers’ perceptions, except for BM who does not 
practise reading. 

4.3 Teachers’ Views of their Students 

Teachers were asked in the survey to describe the performance of the individual case studies in their English 
subject. They were also asked to comment on the individual student’s general comprehension, whether 
students carefully read the written instructions provided in order to complete the set tasks/activities (in a 
general classroom setting or in the computer laboratory). This approach was also used to provide additional 
perspectives on any discrepancy derived from students’ views of their capabilities. The teachers’ responses 
are shown in Table 4 below. 


Table 4. Teachers’ rating of the case studies’ comprehension. 


Teacher’s rating of the individual case studies 

Teachers 

Case 

studies 

Describe students’ 
general comprehension 

Do you think students 
carefully read the 

instructions provided to 
complete the set 

tasks/activities? 

Do you think students 
carefully read the 

instructions provided on the 
computer to complete the set 
tasks/activities? 

TH 

GH 

Excellent. 

GH always does read the 
instructions carefully, and, if she is 
not sure of the set activity she 
seeks assistance. 

Not really appropriate for types of 
activities undertaken in English. 

BH 

Excellent. 

BH always does read the 

instructions, and, if he is not sure 
of the set activity he seeks 
assistance. 

Not really appropriate for types of 
activities undertaken in English. 

TM 

GM 

GM’s general comprehension 
is good. Overall she 

understands and can answer 
questions usually with clear 
evidence and example. 

Usually GM reads the instructions 
provided without too many 
problems. 

GM usually reads the instructions 
provided without too many 
problems. 

BM 

Okay, but only when BM puts 
in the effort to listen. 

At times, but sometimes BM 
forgets to read all information. 

At times, but sometimes BM skips 
through the initial information and 
goes into other activities. 

TL 

GL 

GL’s general comprehension is 
good. Overall, she understands 
but finds it difficult to express 
herself when it comes to 
respond to certain topics or 
issues. 

Reads the instructions but if they 
are still not clear to GL, she does 
not ask for explanation. She does 
the work as she knows best, even if 
it is all wrong. 

Reads the instructions but if they 
are still unclear to GL, she does 
not ask for explanation. She 
briefly works on the set activities 
where she requires regular 
assistance. 

BL 

BL’s comprehension is OK at 
times. There have been issues 
with him to complete any sort 
of work. He tends to get 
sidetracked with other things to 
avoid completing the work. 

BL does not read the instructions 
at all. Sometimes, he copies the 
work from his class mates. 

Does not read the instructions at 
all, sometimes, he does other 
things which are not related to the 
set activities on the computer. 
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Table 4 shows the teachers’ responses were the same for both students in their class. The teachers’ 
description of the case studies corresponds to the students’ answers in Table 3 TH described GH’s and BH’s 
reading and general comprehension skills as “excellent’’. TH commented that both GH and BH always read 
the written instructions carefully and sought assistance if they were not sure of the set activities. Throughout 
my observation period, TH expected the students to handwrite their first drafts until they reached the 
curriculum expectations. I had a discussion with TH about the persistence of the handwriting strategies. The 
response was that "this group is a high achiever group and I would like them to remain at this level by 
ensuring that their vocabulary and grammar are up to standard. Students are not allowed to use computers 
while they are drafting”. Graham et al (2000: 630) support TH’s handwriting strategies to conclude in their 
study that "... handwriting [is] causally related to learning to write” (Graham and Harris 2006: 67 - 68). 

TM was satisfied with GM’s general comprehension as GM usually answered the questions with clear 
evidence and provided examples. GM usually "reads the instructions provided without too many problems”, 
both with and without the aid of the computer. TM described BM’s comprehension as "[o]kay, but only when 
BM puts in the effort to listen”. “..., but sometimes BM forgets to read all information and skips through the 
initial information and goes into other activities”. From the teacher’s comments, BM did not follow class 
instructions and his avoidance of reading the written information suggested that his attention was directed to 
other activities. 

TL responded that GL’s “general comprehension is good”. Overall, TL felt that GL understood what was 
expected of her but she found it difficult to express herself “when it comes to respond to certain topics or 
issues”. TL’s comments were that GL “does not ask for explanation”, even when the instructions “are still 
not clear to her. She does the work as she knows best, even if it is all wrong”. Her teacher’s comments 
recorded in Table 3, TL described GL as “no challenge” very quiet and BL is a “big challenge”, he “does not 
remain in his seat nor pay attention to class instructions”. TL reported that both GL and BL were 
experiencing difficulties reading and writing in English. In line with TL, GL reported that she perceived 
herself as a “good reader” but a “poor writer”, while BL reported that he perceived himself as a “poor reader 
and writer”. 

Narvaez (2002: 158) stated that the causes of individual differences in the comprehension of texts along 
two lines: “reading skill” and “reader knowledge”. Malatesha and Aaron (2010: 317) as well as Myer and 
Wijekumar (2007: 356) stated that vocabulary knowledge is a prerequisite and a critical factor in improving 
reading comprehension. Fielding and Pearson (1994: 62) reported that teachers should set a stage for students 
to succeed at reading, they can “supply ample time for text reading, direct strategy instruction, and 
opportunities for collaboration and discussion”. Readers use their knowledge of text structures to build a 
coherent memory representation, and these structures or relationships are part of their cognitive 
representation. 

Williams’ (2007: 199 - 200) work applies to both GL and BL. Specifically BL had reached a stage where 
his frustration and lack of interest in reading turned to behavioural disruption in contrast to GL, who 
remained quiet in class. The self-confidence of such students, according to McKenna (2012: 15), may decline 
as may their interest in what they are learning. These declines may also lead to lack of motivation and school 
failure. Chen and Lee (2010: 120) asserted that many middle and high school students who have only basic 
reading do not reach the necessary high level of literacy. They avoid reading, like GL and BL, and miss out 
on the opportunities to develop further their language literacy skills because they have not yet become 
autonomous and self-directed readers and learners. 

4.4 Part 3 - Main Uses of Computers 

The survey included open-ended questions about what the students mainly used their computers for, and the 
types of activities they regularly engaged in. In the interviews, I extracted their survey responses and gave 
them their answers as a stimulus for discussion. In the interview, I asked the case studies to elaborate further 
on their survey responses. The main focus was why the students used the computers rather than handwriting 
their school-work. The case studies indicated that accessing the internet was their main computer use. Their 
teachers were also asked about the use of the computers in their English classes. TH responded the use of 
computers in class was minimal. But TM and TL reported they use the computers for research, word 
processing and other software applications. 
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4.5 Case Studies’ Reflection on the Use of the Computer Tools 

This part sought to find out how confident the students felt about composing their English writing without the 
computer tools. When the case studies were interviewed, the questions I asked were based on their survey 
responses in order to validate their answers. I asked them how they felt when the spelling and grammar 
checkers were turned off. Their responses were as follows: 

GH: I do not totally rely on the spelling and grammar checkers. I usually use the dictionary to check for 
spelling errors because it uses the word in a sentence to confirm its meaning and its grammatical 
position. I do not need to use them because my vocabulary and sentence structures are very good. 

BH: I do not totally rely on the spelling and grammar checkers. I usually use the dictionary to check my 
spelling and have my parents to read over the work for both spelling and grammatical errors. I only use 
them for really long document. 

GM: If the spelling and grammar checkers are turned off, I feel that there is something is missing. I rely on 
the red and green squiggly lines to indicate to me that I have misspelt a word or my sentence structure is 
incorrect. When you get use to the computer tools, it is hard to work without them. I suppose we 
become dependent on them. BM: I do rely very much on the spelling and grammar checkers. The red 
and green lines are good things to have because they show me that the underlined words or the 
sentences are wrong. The spell checker helps you for time being but it is not beneficial in a long term. 
GL: I am hopeless. If the spelling and grammar checkers are turned off. I told you earlier that I am a poor 
speller. I always use the spelling and grammar checkers to correct my work 
BL: I do rely very much on the spelling and grammar checkers. But I still do not know which the correct 
word to choose from the list. I always choose the first word on the list because it is correct. I find the 
grammar a bit confusing. 

Reflecting a more on autonomous view of literacy, GL and BL select one of the suggested words 
displayed or ignore the suggestions and leave the word unchanged, whether it is correct or not (Heift and 
Rimrott 2008: 211). An ideological view suggests that the process is not this simple, that other factors may 
intervene in the process of apparently ‘selecting’ the correct option. Ljungdahl (2010: 345) states that some 
students will just choose any word from a list of alternatives on the spell checker if they are poor spellers. 
This applies to BM, GL and BL in the study, where the intended content is lost (Galletta et al (2005: 82 - 86). 
An example was given by BH about his younger sister and his father’s action of turning off the spell checker 
on their home computer. The reason was that his younger sister had started to use abbreviations, had begun 
chatting online, downloading information from the internet, and her spellings had started to deteriorate 

When it came to choosing the right word from the spell checker, GL’s comment was that “..., 1 do rite 
click and click ignore”. She elaborated in the interview that if she chose to use the electronic spelling list, 
“seconds later after I switch the computer off, 1 do not remember how to correctly spell the same word”. She 
always relied on the spell checker, because she felt that she was “a poor speller”. GL claimed that she used 
"the grammar checker all the time”. 

4.6 Frequency and Estimated Time of Computer Use and Reading 

The case studies were asked to include the frequency of computer use and estimate the extent of their home 
computer use and reading per session. They provided estimated times in hours and minutes per session. The 
times were converted to minutes per week to enable comparison of the time fractions that they included. The 
case studies’ responses are shown in Table 5. 


Table 5. Frequency and estimated time of personal computer use and reading per week. 


Frequency and estimated time of PC use and reading per week 

Case 

Studies 

Frequency use of home 
computer 

Estimated minutes per week 
using home computer 

Estimated time in minutes 
reading per week 

GH 

Every second day 

480 

300 

BH 

4 times a week 

360 

420 

GM 

Every day 

840 

20 
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BM 

Every day 

840 

No response 

GL 

Every day 

1680 

No response 

BL 

Every day 

2100 

No response 


The data in Table 5 show the estimated time in minutes per week. GH and BH used their home computer 
four times a week or approximately every second day. GH’s estimated time using her home computer was 
480 minutes (8 hours) per week, while BH used his home computer for 360 minutes (6 hours) per week. GH 
spent two more hours using the home computer than BH. GH’s estimated time spent on reading was 300 
minutes (5 hours) per week, while BH’s estimated time spent on reading was 420 minutes (7 hours) per 
week. BH spent two more hours more on reading than GH. TH reported that both students were “excellent” 
readers and writers. GM and BM spent more time using their home computers of equal time of 840 minutes 
(14 hours) per week than reading the printed text. GL and BL used their home computers “everyday”. GL 
accumulated estimated time of 1,680 minutes (28 hours) per week and BL spent 2,100 minutes (35 hours) per 
week. They did not respond to reading time at home. As the data indicate that GM, BM, GL were indeed 
digital readers. 


5. CONCLUSION 

This paper was divided into three parts. Each part addressed a key element of how the secondary school 
students (the six case studies) used the computer tools to enhance their English writing. Their English 
teachers also reported on how they perceived each individual student’s performance and achievement in their 
classes. The six case studies and their teachers were surveyed and interviewed in order to validate and 
establish any relationship between computer and language literacy skills. The first part presented the profiles 
of the six case studies in terms of their achievements in the ACER formal tests. At this stage, they were in 
Year 8 when they sat the ACER specimen tests (reading vocabulary, reading comprehension and writing). 
As shown in Table 2, the tests results revealed a pattern differentiating the performance of the case studies on 
the tests. The pattern of performance distinguished high (GH and BH), medium (GM and BM) and low (GL 
and BL) scores measured across the three literacy tests. 

Part 2, the teachers (TH, TM and TL) gave their views of the case studies reading and writing as they 
perceived the students in their classes. TH’s perceptions of GH’s and BH’s achievement in reading and 
writing performance were of a high standard. TM reported that when GM and BM produced their work on 
the computer appeared much neater and spelling and grammar were checked, corrected and easily redrafted. 
TL described GL’s and BL’s reading and writing skills as very poor. The data suggested high expectations of 
oneself and self-motivation are the motives to successfully acquire language literacy skills. 

The data in Part 3 indicated that the case studies’ self-ratings on their literacy concepts compared with the 
outcomes of the ACER tests were complemented by their teachers’ perceptions of their performance in the 
English classes. GH and BH read widely (5 to 7 hours per week) and were self-motivated. They were 
influenced and encouraged by their parents who read to them so that GH and BH were read to during their 
growing up years. Both students found the wide reading very beneficial in the development of their literacy 
skills. By comparison the medium achievers, GM read for 20 minutes per week and BM did not read at all. 
The data showed a significant difference in reading time between the high (GH and BH) and the medium 
(GM and BM), where the GM and BM found reading “boring”. Their results in the ACER specimen tests 
showed average to above average scores. The data also revealed that GM and BM sought other sources such 
as the computer tools (spelling and grammar checkers) to enhance their English writing, but only in a non- 
reflective manner. By contrast, GL and BL did not read at all. Their results in the ACER specimen tests were 
below the national average. Both students found reading “very boring”. 

The data suggested that the lack of exposure to interest in reading may have had implications. These 
implications of the study revealed that it is necessary for teachers to teach students how to use the electronic 
spelling and grammar checkers and dictionaries in any language at schools. So the students can use them 
correctly rather than guess and select the words and sentences randomly from the provided list by the tools. 
These tools should become part of the teaching curriculum for all school age students. 
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ABSTRACT 

Drop-outs and delays of graduation is currently a huge problem in adult education. The main reason for the drop-outs and 
delays is usually stated to be the difficulty of combining studies with family and work. This study was based on 
interviews where students studying in the bachelor’s or master’s degree programme were interviewed to find out the main 
reasons for delays and drop-outs, and to ideate and evaluate various ways of supporting students. There were two main 
types of reasons for delays and drop-outs: reasons relating to difficulties in combining personal and working life with 
studies, and problems related to the institution, e.g. challenges related to the theses, teaching methods, and lack of 
guidance counselling. According to the study, there are several opportunities to boost students’ graduation: a named 
personal guidance counsellor during the studies and for the thesis, regular follow up of studies, individual learning paths 
counselled by a guidance counsellor, prior competence recognition, flexible methods to study for example by offering E- 
learning courses. 


KEYWORDS 

Adult education, drop-out, support, guidance counselling, online tools 


1. INTRODUCTION 

Adult education is increasingly important because of the quick pace of technical development, changing 
demands for skills and with the general raise in the level of education. Studying towards a degree while 
working and often with family responsibilities is demanding. In practice, this leads to long times for studying 
and dropping out. Student may get some benefit of the courses they take and skills they acquire even without 
reaching the degree, but overall, it is a problem for all parties involved, the student, the school and the whole 
society. 

Drop-outs and delays of graduation have been studied a lot and there are several frameworks to categorize 
the factors. First, a framework developed by (Jordan et al., 1994; Watt & Roessingh, 1994) defines three 
factors for drop-outs; push, pull, or fall out of an institution factors. In push-out factors, an institution is 
active and the factors include poor academic performance and disciplinary problems. Pull-out factors refer to 
out-of-institution factors like work or family reasons, or illness. The fall out factor mean that a student does 
not show significant academic progress and in a way a student just disappears. During several decades the 
pull factors are depicted as the highest factor for dropout (Doll et. al, 2013). Second, Rumberger & Lim 
(2008) defined two groups of factors to explain reasons for drop-outs; individual characteristics of students 
and institutional characters of their families, schools and communities. Solutions, such as more flexibility in 
schedules (e.g. Johnson et. al, 2009), and prior learning assessment and recognition (e.g. Aarreniemi- 
Jokipelto, 2014) have been suggested earlier to boost graduation. 

In this study we wanted to find out the reasons why adult students studying either in the bachelor’s or 
master’s degree programme in business information technology at HAAGA-HELIA University of Applied 
Sciences have proceeded slowly in their studies or dropped out. The study was aimed at students who had 
already proceeded long in their studies. We also ideated a number of tools and practices that could be helpful 
to students to get them to complete their studies, and wanted to get students evaluations of how beneficial 
these would be from their perspective. We also asked the students to give their own suggestions of ways 
support graduating. 
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2. RESEARCH METHODS 
2.1 The Method 

We chose telephone interviews as the way to gather the information of the students. We made a questionnaire 
that was filled together with each student during one phone call. Even though this method limited the number 
of students we could include in the study, it gave us the opportunity to talk to people and get a better picture 
of their situation and decisions that would have been possible for example by using a web survey. 

The questionnaire consisted of seven main points: 

1 . Motivation for studies 

2. Life situation and changes in it 

3. Planning and guidance counselling of studies 

4. Content and quality of studies 

5. Thesis 

6. Main reasons for delay or dropping out 

7. Feedback on tools and methods for supporting studies and thesis 

We interviewed 38 persons who were studying on extended period or had quit their studies during 2013 
or later (Table 1). We wanted to interview sUidents who had studied for several periods and gathered credits, 
both those who had started their thesis and those who had not. There were 12 interviewees (32 % of all) 
aiming at master’s degree, and 26 or 68 % aiming at lower degree. This proportion reflects on the distribution 
of adult students between higher and degrees. Only two master’s degree students (17 %) had dropped out 
whereas 12 or 46 % of bachelor’s degree students had dropped out. 

The master students had in average 55.4 credit points (60 credit points required + the thesis worth 30 
credit points). The bachelor students had in average 160.2 points (195 + 15 credit points required). Typically 
the students had been present at the university for 10 semesters. 

The master’s students originated from 6 different starting groups, but half of the interviewed students 
came from one particular starting group. The bachelor students came from 16 different starting groups or 
study programs, with the maximum of three respondents from one group. 

We both made 19 telephone interviews. We took notes by hand during the discussion and wrote complete 
notes on computer after the discussion. We divided the analysis of the results so that one of us analysed the 
reasons for quitting and aspects relating to study planning and guidance counselling, as well as issues relating 
to the thesis, and the other one looked at the feedback on the proposed tools and methods to support sUidents. 
After the preliminary analyses, we looked at each other’s parts and complemented it with the insights we had 
gained but that had not been included in the preliminary analysis. 

Table 1. The number of interviewees divided into groups based on the degree and whether they had started their thesis 



Extended time (code) 

Drop out (code) 

Total 

Higher degree, thesis ongoing (Al) 

7 

1 

8 

Higher degree, no thesis (A2) 

3 

1 

4 

Subtotal higher degree 

10 

2 

12 

Lower degree, thesis ongoing (B 1 ) 

12 

5 

17 

Lower degree, no thesis (B2) 

2 

7 

9 

Subtotal lower degree 

14 

12 

26 

Total 

24 

14 

38 
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Table 2. The average number and standard deviation of semesters present and absent and credit for the interviewees 



Semesters present 

Semesters absent 

Credits 

Credits/ active semester 

Eligher degree (A1+ A2) 

Average 

9.5 

0.5 

55.4 

6.0 

Standard deviation 

2.6 

0.8 

15.1 

1.5 

Lower degree (B 1+B2) 

Average 

10 

1.2 

160.2 

17.0 

Standard deviation 

3.8 

1.8 

50.1 

4.6 


Since the number of interviewees was relatively low, particularly because they needed to be divided into 
four subgroups based on their situation, we have treated the material as qualitative. The results give 
information about the aspects that need to be taken into consideration when developing services for adult 
learners. 


3. REASONS FOR DELAYS AND DROP-OUT 

Table 3 illustrates the results of the questions related to the reasons for delays and drop-out of studies for the 
master’s degree students. A1 in the table refers to students who are studying their master’s degree and had 
started their theses, and group A2 refers to students who had not yet started their theses. 

Table 3. The most important reasons for the delay or dropping out in the group of master’s degree students (number of 
students giving the reason (A1 thesis ongoing; A2 thesis not started)) 



Most 

important 

Second in 
importance 

Third in 
importance 

Personal life situation and /or changes in it 

- 

1 (Al), 1 (A2) 

1 (Al) 

Combination of work and studies did not work 

3 (Al) 

- 

- 

Changes in working situation 

1 (Al), 1 (A2) 

2 (Al) 

- 

Interest, motivation ended 

1 (A2) 

- 

1 (Al) 

Career changes 

1 (Al) 

- 

- 

Problems related to the theses 

5 (Al), 1 (A2) 

2 (A2) 

- 

Group work based learning 

1 (A2) 

- 

- 


Over half over the group A1 students named problems related to the theses as the most important reason 
for delay or drop-out in their studies. Some students had faced problems in combining work and studies. If a 
second or third reason was given, they were related to the personal life situation, changes in working 
situation, or lack of motivation to study. From the institution point of view problems relating to the theses 
and combining work and studies are worth investigating more. 

When asked, students named several types of problems in the context of theses: lack of a subject, the 
viewpoint of the thesis was missing, and changes at work. It was common to have problems to find a subject 
for the thesis. There were many cases where the student had completed the other studies and only the thesis 
remained. There were also students who had managed to find a subject for their thesis, but then faced 
problems in the writing process. Two students stated that the viewpoint of their thesis was missing, and they 
were not able to continue. There were a few students who had changed their working place, and they were 
forced to change the subject of their theses. Also, a new job often required a lot of effort and energy, which 
weakened the opportunities for studying. Organisational changes had also caused problems: in one case, the 
planning of the thesis had to start over with a new boss; in one case, the responsibilities were changed at the 
workplace and the student was not any more in charge of the area to which the thesis topic was connected. 
The economically tough times during the recent years have in general put a lot of stress in working places, 
which has reduced the time and energy on extra activities like studying. 

Our questionnaire also included questions about guidance counselling during the studies and follow-up of 
the study process. The interviewed students were from several different programs, and there was a lot of 
variance between programs on whether the student had got a guidance counsellor, whether a guidance 
counsellor had been active, and whether the studies had been followed-up and assessed by a guidance 
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counsellor during the study process. It is clear that in cases where the reason for the delay or drop-out has 
been a problem to find a subject for a thesis, guidance counselling would have been required. It would have 
been a guidance counsellor’s duty together with a student and a working life representative to look for the 
subject for the thesis. Also, if the problem has been the viewpoint of the thesis, the guidance counsellor 
would have been able to solve the problem together with a student. In few cases, there had been problems 
already in the earlier phases of the study process, and students would have needed their studies to be 
guidance counselled, followed up and assessed. Also, if the working place has changed and the student needs 
to look for a new topic for the thesis, it is important that a guidance counsellor is available. 

To summarize, both during the study and the thesis process each student should have a named guidance 
counsellor, who is in contact on a regular basis, and whom the students can contact whenever required. Few 
students stated that they would have needed someone to kick them to proceed in their studies. 

Table 4. The most important reasons for the delay or dropping out in the group of bachelor’s degree students (number of 
students giving the reason (B 1 thesis ongoing: B2 thesis not started)) 



Most 

important 

Second in 
important 

Third 

important 

Forth 

important 

Personal life situation and /or 
changes in it 

6 (Bl), 
2 (B2) 

2 (Bl), 
1 (B2) 

1 (B2) 

" 

Combination of work and 
studies did not work 

2 (Bl), 

3 (B2) 

4 (Bl) 

1 (Bl), 

2 (B2) 

1 (B2) 

Changes in working situation 

3 (Bl), 
2 (B2) 

1 (Bl) 



Interest, motivation ended 

1 (Bl) 

2 (Bl), 
1 (B2) 

2 (Bl) 


Career changes 


2 (B2) 

- 

- 

Problems related to the theses 

5 (Bl), 
1 (B2) 

1 (Bl) 

1 (Bl), 
1 (B2) 

- 

Lack of guidance and 
counselling during the studies 

1 (Bl) 

1 (B2) 

1 (B2) 

- 

Lack of guidance and 
counselling during the theses 
process 



1 (Bl) 


Group work based learning 

- 

1 (B2) 

- 

- 

Degree does not have value 

- 

- 

1 (Bl) 

- 

Uncertainties in assessment 

- 

- 

1 (Bl) 

- 

Financial aid ended 

1 (B2) 

- 

- 

- 

Studies is a hobby, no need to 
proceed fast 

1 (B2) 

- 




Table 4 presents the reasons for delay and dropping out for bachelor’s degree students. In Table 4, B1 
refers to students who are studying their bachelor’s degree and who have started their thesis and B2 refers to 
students who have not started their thesis. In group Bl, the most important reasons for the delay and drop-out 
were the following: 

• Personal life situation and/or changes in it 

• Problems related to theses 

• Changes in working situations 

• Combination of work and studies did not work 

When comparing the groups Bl and B2, the difference is that B2 has not named so many problems 
related to theses, because they have not yet started their thesis. Otherwise they seem to have faced same kinds 
of problems as the other group i.e. to study aside with family and work can be challenging. The only how we 
can solve these kinds of problems are to increase flexibility of studies and to provide different methods to 
study, and not to give group works as the only study form. Both in the master’s and bachelor’s degree groups 
lack of motivation was also stated as a reason for delays and drop-outs. To improve motivation flexibility of 
studies, meetings with peers and inspiring visiting lecturers from outside of the university was suggested. 
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As mentioned earlier, the interviews included also questions about guidance counselling during the 
studies and following up on a study process. When comparing the bachelor’s degree groups to the master’s 
degree groups there were much more students in the bachelor’s degree group, who have not had a guidance 
counsellor. Neither students’ studies had been followed up and assessed during the studies, which can be 
assumed to have an effect on the delay or drop-out of the studies. 

There seemed to be a kind of “survivor culture” among the interviewees. Some students told about 
challenges relating to their studies, but still they had not been in contact with a guidance counsellor or the 
institution, because they felt they need to manage on their own. Few students stated that because we are adult 
students, we need to manage the studies alone. There were several students who had not had any guidance 
counselling during their studies. If a student had a guidance counsellor, it did not mean that they had been in 
contact with her/him when assistance was needed. A student stated that he had bothered a guidance 
counsellor twice with his study problems. Generally, the threshold to ask for help seemed to be high among 
many students. Although, there were students whose studies had not been guidance counselled or followed 
up at all, they did not necessarily mention it as a problem. They just seemed to have accepted the practice that 
they are alone with their studies. There were several students whose graduation would have benefitted from 
guidance counselling meetings and regular follow up of studies. Few students stated that they would have 
needed someone to kick them to be able to proceed faster in their studies. A student suggested help in time 
management skills to be offered in the beginning of the studies. 


4. TOOLS AND PRACTICES FOR SUPPORTING STUDIES 

Our questionnaire included a list of 1 1 ideas relating to how adult learners could be helped to reach their 
studying goals. The students were asked to evaluate the ideas on a four level scale: very beneficial, somewhat 
beneficial, no benefit, and detrimental. Many respondents also gave additional points of view regarding the 
ideas. The students were also given the opportunity to make their own suggestions as to what would be 
helpful to them. 



Figure 1. The percentage of respondents who evaluated the proposed tools as very beneficial, or as have no or detrimental 
impact on their study progress. The rest of the respondents gave the rating somewhat beneficial, but this is not shown in 

the figure 
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The most highly rated idea was the communication app: an application that would make it possible for the 
student, the supervisor at the workplace and the guidance counsellor at the university to communicate and 
stay up to date with the progress of the work. Following viewpoints were presented: 

• If the app would help that problems become acknowledged early, then this would be good. 

• It would be good if there is something that would force the three parties come together and talk 
about the thesis. 

The following two, a bit more critical viewpoints were also made: 

• Adult learners do not need something like this. 

Something like Dropbox would be enough - there the guidance counsellor would immediately see the 
new version of the work and could comment directly into the text. 

The Contact app was a related idea: the idea included various ways of easily getting in touch with the 
guidance counsellor such as chat and the opportunity to book a meeting time from a list of available meeting 
times. Many of those who did not regard the application as useful were of the opinion that email is enough. 
Some respondents pointed out that they mostly study during evenings and weekends and wondered whether 
the guidance counsellor could be available at such times. 

More interactive eLearning material that would make it possible for the students to study at their own 
pace and when they have time, was regarded as very beneficial by 42 % of the respondents, the second 
highest rating. Following three viewpoints were given in connection to eLearning: 

• Computer languages would be very suitable for eLearning. 

• Particularly good for those who live further away from the school. 

• The most central and critical courses need to be taught face-to-face, but eLearning should be 
good for the rest. It would be good to be able to complete courses when one suddenly gets time 
for the studies 

One student had been inspired to start studying after having participated in an excellent eLearning course 
at her work place, and had been disappointed as there had not been such eLearning courses at all. 

Mobile learning was described in the following way: learning material that is split into small units that 
can easily be studied on smartphone or tablet during small breaks or when commuting. This idea raised 
mixed reactions and even strong emotions among the respondents. 36 % thought it would be very beneficial 
to them. For example, it was seen as a potential platform for small weekly assignments. The same percentage 
did not see any potential benefit; one respondent even regarded it as detrimental. Those who did not regard 
this as a good idea expressed following views: 

• I do not want to use a device all the time; time in a bus is good for recovering and incubating. 

• Is a smartphone a real tool for learning? 

• To learn, I need to concentrate properly, which is impossible during small breaks. 

• Smartphone is good for checking facts, but not suitable for real studying and learning. 

Automatic notifications sent to the student’s mobile were another idea that raised mixed opinions: 30 % 

regarded it as very beneficial, and 31 % as not beneficial or detrimental. Those who did not see any benefit 
motivated their answers with 

• Adult learners do not need a “nanny”. 

• Adults need to learn to plan and manage their timetables by themselves. Automatic notifications 
just teach students to rely on others the wrong way. 

The importance of time management came up in various ways during the interviews, as well as the 
importance of being able to find all the necessary information of courses and events easily in one place. 
Offering time table information so that people can easily integrate this information to their preferred 
calendars and time management apps in beneficial. 

Study progress report, meaning better and clearer information of how one’s studies are progressing in 
comparison to one’s personal study plan, lightweight reporting in the form of a weekly “smiley” from the 
student to the guidance counsellor indicating how the thesis is progressing, and more online student 
collaboration were regarded as very beneficial by around 20 % of the respondent. About half of the 
respondents found these ideas as having no benefit or even being detrimental. 

The idea of lightweight reporting raised most mixed opinions. A slightly surprising finding is that all 
students who thought this would be a very beneficial feature were aiming at the bachelor’s degree, where the 
thesis is smaller than in master’s degree. Two respondents, both of them aiming at the master’s degree found 
the idea detrimental. Many criticized the proposed weekly rhythm. It was suggested that the rhythm should 
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be defined individually depending on the phase of the work, or the reporting should be made less frequently, 
for example, once a month. 

Online student collaboration and communication was not regarded as very beneficial, particularly when 
making the thesis. Two respondents emphasized the role of the guidance counsellor in online environments: 
without a guidance counsellor leading the discussions, the discussions tend to remain very shallow and do not 
contribute to learning. 

More flexibility in studies was regarded as very beneficial by 34 %. Many commented that the studies are 
already very flexible and did not see many realistic opportunities for chances in this area. 

Stricter timetables and deadlines were seen as the most detrimental idea with 39 % expressing that 
opinion. The combination of working and studying and often also with a family with under aged children 
makes it difficult and even impossible for follow strict timetables, and would only lead to quitting. Keeping 
track of individually agreed timetables and deadlines and making an effort to meet them was however seen as 
important, and this would support completing the studies. 

The least beneficial idea was better paper based learning materials such as books and handouts. The 
students were afraid of the costs and accessibility of books, and therefore preferred getting the study material 
in electronic format, such as PDF files. These are also easy to keep as a reference library for later. 

The students were also asked to make their own suggestions of tools and practices that would be 
beneficial to them and following ideas were shared: 

• Ease of use in accessing school information systems so that small hurdles would not lead to 
quitting 

• Comprehensive information of courses and registration to courses online 

• Clear course descriptions so that it is easy to evaluate if the course topic and level are 
suitable 

• Clear information of what to do when one wants to start the thesis, and what to do when it is 
completed 

• Study groups of 3 to 4 students with related thesis topics could meet and comment each 
other’s work also without advising teachers 

• Support to buying a personal computer that would fit the school requirements 

• Personal messages from the guidance counsellor to keep up motivation and to indicate that 
the student has not been forgotten 

• Live online lectures 

Many of these suggestions deal with getting information easily. The time that the students can devout to 
studies and familiarise with the school is very limited so being getting information and up-to-date with all 
matters at school is more difficult than to fulltime students. Tiny hurdles can lead to bigger motivational 
problems leading to delays and even dropping out. 


5. CONCLUSIONS AND DISCUSSION 

Before the research, the reasons for delays and drop-outs of studies were assumed to be mainly beyond the 
control of the institution i.e. difficulties to combine studies with work and family. However, according to our 
research, there are several ways for the institution to boost graduation by changing its own action. Over half 
of the students who were studying their master’s degree and several bachelor’s degree students had faced 
problems related to making the theses. The mandatory requirement of the master’s level to link the student’s 
working place to the theses had caused challenges and delays. According to the interviewees, a possibility to 
start the theses already at the beginning of the studies would have benefitted them. It would have given time 
to plan and gain ideas from the studies. On the other hand, many had started to plan their thesis early, but 
then, changes at the working place had ruined the planned thesis. There was also a request to receive 
assistance from the institution at the beginning of a theses process, i.e. a guidance counsellor of the institution 
to negotiate about theses with a student’s employer together with a student, a guidance counsellor to assist in 
choosing the topic of a thesis, and to assist choosing the viewpoint of a thesis. This type of guidance 
counselling needs to be in person. Assistance from a guidance counsellor would also have been needed 
regarding research methods and the structure of the thesis. There were students who had dropped-out due to 
the unsolved problems in making their theses. 
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According to the interviewees, it is common that prior learning assessment and recognition (PLAR) is not 
utilized consistently in the programmes. Some students told that they had participated in courses on topics 
where they already knew everything, because it was less work than passing it by showing prior competences. 
The experience of the students was that the requirements and practises for PLAR varied from course to 
course making it hard and unpractical to use. This had partly reduced motivation and delayed graduation. 

Another mentioned reason for delays and drop-outs was the teaching methods of the studies. All the 
students were adult students, and some of them stated that mandatory group works had delayed their studies, 
even causing dropping out. A student stated that she had tried to negotiate a tailored way to pass a course, but 
not succeeded, which had resulted as a drop-out. She stated that she could have been able to study the subject 
tailored, but was unable to participate to several group meetings. Therefore, according to the interviewees 
flexible teaching and learning methods can also boost graduation. Generally in the research PLAR was stated 
to motivate students, but it also to enable shortening the studies. In this case it could have boosted the 
students’ graduation in the programmes. Also, individualized learning paths and flexible learning methods 
would speed up graduation. In some cases, students would have expected a stronger role from the lecturer to 
make sure that the students were really learning the topic. 

Yasmin (2013) has aimed at finding ways of predicting the student categories that are at higher risk of 
leaving open and distance learning courses prematurely. According to the study, students who are married 
employed, or over 25 years old are most likely to drop out from the course. However, he suggests not 
excluding or discouraging those students at higher risk from taking distance education courses. Rather, the at- 
risk students should be leveraged for developing and refining policies and processes that effectively serve 
these students. In the study conducted at HAAGA-HELIA the students were adults and employed students, 
which were noticed to be the at-risk students already before the study. We agree with Dr Yasmin that the 
institution should develop new processes and polices to serve these students. In practise this means flexible 
teaching methods, providing different paths to be followed to gain the competences defined in the 
curriculum, but also constant and in person guidance counselling. 

A study conducted by Street (2010) attempted to find factors influencing a student’s decision to drop-out 
or persist in higher education distance learning. According to the literature review, several common themes 
have emerged across the research including significant external factors such as course factors and support, 
person factors such as self-efficacy and autonomy, and academic factors such as time and study management. 
Street has divided course factors into relevance and design. In our study personal factors were not studied, 
but it was noticed that the students are a heterogeneous group with different needs and requests relating to the 
teaching and learning. According to our research, external and course factors have huge importance. 
Instructional design choices were seen crucial from students’ persistence of continuing studies. Also, course 
factors: support and guidance counselling were crucial factors from continuing studies point of view. 

E-learning tools to study, guidance and follow up of studies can benefit students’ graduation, but the tools 
cannot replace a personal guidance counsellor. Interactive eLearning materials that would make it possible 
for the student to study at their own pace and when they have time were regarded as beneficial by 
interviewees. In addition to that, mobile solutions can benefit some of the students in their studies, but there 
was also a group of students which felt that mobile solutions are totally appropriate for them. We can see that 
the adult students are a very heterogeneous group, and their needs and preferences vary a lot. This means that 
more individual learning paths with alternative ways of carrying out their studies should be available. When 
developing the study offer for adult students, the time efficiency and independence of time and location 
should be given high priority. However, keeping up the motivation needs human contacts, personal guidance 
counselling, peer student meetings and flexible teaching methods. E-learning can be utilized to provide 
efficient learning possibilities, but also to utilize interaction with guidance counsellor and peer students. 
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ABSTRACT 

Many studies suggest factors that might influence online learning and assessment, but most have not been empirically 
tested. We use survey data from 91 students to investigate what factors influence students’ satisfaction with online 
assessment, and overall student satisfaction with the learning management system - Moodle. The survey questionnaire 
was sent anonymously to all students who took the online or hybrid course sections at a private Midwest university in 
summer 2013, as well as to all instructors who taught those courses. The result showed that overall student satisfaction 
with online learning is significantly affected by how the course is organized and how the content is sequenced, the ease 
with which students can complete assignments, and the use of the system to engage students with content. 
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1. INTRODUCTION 

According to Allen and Seaman (Allen & Seaman, 2013), the Sloan Online Survey found that in 2002 less 
than half of U.S. higher education institutions reported online education as being critical to their long-term 
strategy. However, in 2012, 70% of the surveyed academic leaders saw online learning (or e-learning) as 
critical to their long-term strategy. The 2012 report showed that there were 6.7 million students (32% of the 
total student population in the U.S.) taking at least one online course. Furthermore, 77% of academic leaders 
rated the learning outcomes in online education as the same or superior to face-to-face instruction. Thus, on- 
line learning is here to stay and we have to learn to use it effectively. Higher education continues to evolve, 
including recent increases in the number of courses offered fully and partially (hybrid) online. In order to 
offer these courses, it is important to have an effective technology platform to support the online class 
environment and activities. Many new technologies are used to support online education, particularly 
learning management systems (LMS's), which serve as the core technology platforms for the online 
environment. Some popular systems include Blackboard, Moodle, Sakai, Desire2Learn, Canvas, and 
eCollege. Moodle as an open technology platform is emerging as the most cost effective solution. Hence, 
our study uses Moodle and we believe our findings can be applicable to all LMSs. 

LMS systems support many important functions for effective online education, including facilitating 
instruction, assessment, and course administration as well as providing a new means for communication with 
students. To create a strong foundation for successful online education, it is critical for universities to 
measure how well the systems they use deliver on the key functions and support the online learning 
environment to serve their core academic mission. The use of new technologies and the application of new 
educational models need to be supported by systematic redesign of the processes at both the institutional and 
educator levels (Georgouli, Skalkidis, & Guerreiro, 2008). Hence, there is a need to understand factors that 
affect student satisfaction with online education and the effectiveness of the platform as an effective learning 
and communicating tool. 
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2. LITERATURE REVIEW 

Online learning is rapidly growing in society today; however, there are many variables that help explain the 
contrasts between online and traditional face-to-face learning as well as how they influence student 
satisfaction. According to Sun et al. (Sun, Tsai, Finger, Chen, & Yeh, 2008), some of the factors that 
influence student satisfaction with online learning systems include: computer/technology anxiety on the part 
of the learner, instructor attitudes toward e-learning, course flexibility and quality, perceived usefulness of 
the LMS and ease of use, and the diversity in assessment methods. Student satisfaction with online learning 
is a significant predictor of learning outcomes, and could be influenced by course structure, instructor 
feedback, students’ self-motivation and learning style, interaction and instructor facilitation (Eom, Wen, & 
Ashill, 2006). According to Bell and Federman (Bell & Federman, 2013), online learning should provide 
content, immersion, interactivity, and effective communication. A comparison study conducted by Summers, 
Waigandt and Whittaker of student achievement and satisfaction in an online versus face-to-face statistics 
course found that although students taking the online course learned as much as students in the face-to-face 
course, online course students were less satisfied with the method of delivery as compared to students take 
the course face-to-face (Summers, Waigandt, & Whittaker, 2005). By focusing on the factors that affect 
student satisfaction, LMSs can be improved and implementation strengthened to increase the learner 
satisfaction and retention of the material. 

Online learning presents an array of challenges and issues to instructors who depend on technology 
systems to completely deliver and support their courses. There are concerns about the validity, practicality, 
and reliability of online student assessment (Dermo, 2009), as well as concerns about the security (Alwi & 
Fan, 2010) and academic honesty (King, Guyette, & Piotrowski, 2009). Stacey and Wisenberg noted the 
importance of key variables, such as length of time teaching face-to-face and online, overall teaching load, 
class size, and institutional context that affect faculty members’ motivation and attitudes towards moving 
from traditional to virtual classrooms (Stacey & Wiesenberg, 2008). 

Naveh et al. (Naveh, Pliskin, & Tubin, 2010) found significant correlation between LMS use and student 
satisfaction in terms of similar organizational variables, which included class size, course content, instructor 
status, and the existence of forums. Given that a properly implemented and supported online learning system 
can help students' and instructors' initial online experiences run more smoothly and ease the transitions from 
face-to-face classrooms to virtual learning, it is important to study the factors that influence successful 
adoption of online learning. A LMS is one of the most representative e-learning applications. The LMS is 
used for online education as well as to supplement face-to-face courses. It is commonly used to post a 
course’s syllabus and announcements, homework assignments and projects, and lecture notes and slides for 
students to access online (ODCD, 2005). There is a strong movement toward open-source solutions (away 
from proprietary software) for e-learning applications (Coppola & Neelley, 2004). Open-source software 
provides flexibility to combine languages, scripts, learning objects, and lesson plans without the steep cost of 
proprietary packages (Williams, 2003). Moodle is an example of such an open-source LMS, and is the 
platform that is the subject of our study. 

According to Georgouli et al. (Georgouli et al., 2008), online content should be complemented and 
enhanced through activities (e.g., blog, discussion board, etc.) to facilitate self-learning. Additionally, 
according to Selim (Selim, 2005), instructors of online courses need to have a positive attitude towards the 
technology, and the students should have competent computer skills. Consequently, one of the biggest 
challenges with online learning is that students and instructors may lack the knowledge or skills required to 
use an online LMS for the first time. A poor first experience can scare students away from online learning in 
the future (Georgouli et al., 2008). 

Regardless of whether it is delivered online or face-to-face, effective teaching involves providing an 
opportunity for student-faculty interaction, active learning, and prompt feedback. The research findings 
suggested that online education can be a superior mode of instruction with timely, meaningful instructor 
feedback of various types (Eom et al., 2006). Yet, in the online environment, interactions, learning, and 
feedback often require unique strategies due to the challenges presented by technology-mediated teaching. 
Students take online courses are expected to take greater control of their learning process and be more active 
in stimulating their peers’ learning, therefore facilitation of online learning emerges as an important role in 
guiding these student-centered approaches (Baran, Correia, & Thompson, 2011). 
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Moreover, as the hierarchy in the online environment is flattened with more distributed power and control 
(Schmm & Hong, 2002), instructors are expected to adopt more facilitative approaches in creating learner- 
centered online classrooms (Salmon, 2004; Smith, 2005). While there is still a strong focus on the 
responsibilities of instructors in online courses, the instructor moves from being at the center of the 
interaction or the source of information to the "guide on the side," which implies that instructors design, 
organize, and schedule the activities and learners assume greater responsibility for their learning by 
coordinating and regulating their learning activities (Anderson, Rourke, Garrison, & Archer, 2001; Berge, 
2009) (p. 429). 

The notion of teaching online requires the development of new skills and sets of pedagogies has led 
researchers to study the roles that online instructors take in online education environments (Anderson et al., 
2001; Berge & Collins, 2000; Goodyear, Salmon, Spector, Steeples, & Tickner, 2001; Graham, Cagiltay, 
Lim, Craner, & Duffy, 2001; Guasch, Alvarez, & Espasa, 2010; Salmon, 2004). When designing online 
courses, it is important to consider the format and effectiveness of student assessment. According to Reeves 
(Reeves, 2000), there are different approaches to incorporate alternative assessments into online education, 
such as cognitive, performance, and portfolio assessments. The use of assessment rubrics for student 
assessments would make assessment activities more reliable and valid to provide more formal measures of 
achievement (Oncu & Cakir, 2011). 

2.1 Hypotheses 

Although some studies suggest factors that might influence online learning and assessment, they were not 
focused on the influence of LMS on students’ satisfaction with online learning, and rarely connect the 
assessment with students’ satisfaction directly. In particular, literature in online assessment is mostly 
theoretical, and lack of empirical testing. Our research therefore sought support for the following two 
hypotheses using empirical survey data: 

HI: Student satisfaction with online assessment is affected by their experience with the LMS, 

the ease of accessing the functionalities of the LMS, and the interaction between instructors and students. 

H2: Overall student satisfaction with online learning is affected by the extent of online 

assessment activities, experience with the LMS, the ease of accessing the functionalities of the LMS, and the 
interaction between instructors and students. 

3. METHDOLOGY 

The survey questionnaire was sent anonymously to all students who took the online or hybrid courses at a 
private Midwest university in summer 2013, as well as all instructors who taught those courses. All 
questions from the questionnaire are included in the appendix. For testing our hypotheses, data analysis is 
focused on the data collected from students. Additionally, we include summary responses from the faculty 
survey in the Conclusion section; these responses help to build the connections between the student and 
faculty perspectives. 

3.1 Sample 

The sample for the study included all 26 online or hybrid courses taught in summer 2013. Approximately a 
quarter of the students enrolled in accounting classes in the College of Business, another quarter was from the 
College of Liberal Arts and Sciences, and another quarter was from the College of Education. In total, 319 
students were enrolled in those sections, and we received responses from 9 1 students, for a response rate of 
27.7%. There were in total 22 instructors who taught the online or hybrid course sections, and we received 
responses from 17 of them, for a response rate of 77.3%. 
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3.1.1 Variable Definitions 

A majority of the students agreed or strongly agreed with the following questionnaire statements: (1) Moodle 
is straightforward and easy, (2) The organization and sequence of course was easy to navigate, (3)1 am able 
to complete class assignments in Moodle, and (4) In the majority of my courses, I interact and do things with 
content rather than read/view the content. We were reassured by the fact that our participants had good 
experiences with Moodle and relied on it to complete their class assignments, including doing things beyond 
simply viewing documents. Further, we conducted a factor analysis for all the items and found that there 
were four components and five distinct constructs. Table 1 shows that these four items loaded together on a 
single factor in Component 1, which we refer to as the “Moodle Experience” labeled (M Exp). In addition, 
the Component 1 factor structure shows that there is another distinct factor that captures the experience with 
accessing materials (refer to as M_Access). M_Access and the other constructs from our confirmatory factor 
analysis, provide us the following results shown in Table 1. 


Table 1. Confinnatory Factor Analysis - Principal Components Analysis with Varimax Rotation 


Items 

Component 1 

Component 2 

Component 3 

Component 4 

Orient M 

0.076 

0.108 

-0.007 

0.944 

Adequate Orient 

-0.013 

-0.018 

-0.069 

0.940 

View Grades 

0.429 

0.209 

0.656 

-0.126 

AccessFeedback 

0.268 

0.259 

0.824 

0.007 

Turnitin Comments 

0.172 

0.288 

0.832 

0.002 

ViewFolder 

0.813 

0.349 

0.213 

-0.014 

View Web 

0.853 

0.222 

0.251 

0.021 

Complete Assign 

0.847 

0.251 

0.189 

0.047 

Complete Turnitin 

0.749 

0.301 

0.142 

0.137 

View Panopto 

0.612 

0.418 

-0.022 

0.020 

Take Quiz 

0.584 

0.181 

0.316 

0.263 

News Forum 

0.496 

0.469 

0.356 

-0.102 

Discuss M 

0.575 

0.318 

0.492 

-0.051 

Email M 

0.427 

0.558 

0.440 

-0.051 

Chat M 

0.282 

0.694 

0.361 

-0.003 

WikiM 

0.194 

0.790 

0.298 

0.148 

Blog M 

0.282 

0.822 

0.192 

0.043 

AdobeM 

0.289 

0.724 

0.127 

0.038 

Straight Forward 

0.662 

0.220 

0.417 

0.003 

Course_Seq_Navigate 

0.751 

0.116 

0.365 

0.010 

CompleteAssign 

0.743 

0.121 

0.223 

-0.036 

Interact Do Things 

0.491 

0.296 

0.063 

-0.053 


Rotation converged in 6 iterations. Four factors extract 71% of the variance. Kaiser-Meyer-Olkin Measure of Sampling 
Adequacy = 0.914, Approx. Chi Square = 503.188. Bartlett’s Test of Sphericity df= 325, Sig. = 0.000 


In Table 1, Component 1 has six questions about access to course content and two 
communication/collaboration questions loaded together on a single factor, which is labeled as M_Access. 
The remaining five communication and collaboration items loaded together on a single factor as Component 
2, which we labeled as M_Comm. These constructs included items such as messaging, using IM/e-mail, chat, 
wiki, blogs, and Adobe Connect. The M Assess construct consists of three items shown as Component 3 in 
Table 1, and are related to conducting student assessment in Moodle. The first item measures the ease with 
which one can view grades, the second item measures the ease of accessing feedback on assignments, and the 
third question measures the ease with which one can view comments provided for assignments submitted via 
the Turnitin function of Moodle. Component 4 captures the adequacy of training provided to students, and 
we label it as “Train”. 

Table 2 shows that the “Train”, “M_Assess”, “M_Access,” and “M_Comm” variables have high 
Cronbach Alpha scores (i.e., 0.91, 0.89, 0.95, 0.96, and 0.83 respectively), which is well above the acceptable 
reliability score of 0.70 (Nunally, 1970). Hence, we conclude that the scales used are reliable. Our main 
dependent variable is a single questionnaire item measuring the overall satisfaction with Moodle. 
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Additionally, we also use M_Assess as a dependent variable to examine which factors affect online 
assessment. Since the university offered voluntary Moodle training opportunities for its faculty and students, 
we control for familiarity with it using two questions, which also loaded on a single factor (labeled as 
“Train”). Additionally, we control for students' gender, class standing, the college to which they belong, and 
their GPA. 


Table 2. Descriptive Statistics 


Variables 

Mean 

Std. Deviation 

No. Items 

Cronbach Alpha 

Overall Satisfaction 

2.20 

1.03 

1 


Train 

1.81 

0.38 

2 

0.91 

M_Assess 

3.54 

1.15 

3 

0.89 

M_Access 

3.94 

0.92 

8 

0.95 

M_Comm 

3.32 

1.03 

5 

0.96 

M_Exp 

3.90 

0.80 

4 

0.83 

Gender 

1.77 

0.42 

1 


Standing 

2.91 

0.97 

1 


GPA 

5.46 

0.87 

1 


College 

3.44 

1.85 

1 



Labels are described in the Appendix. Sample size used to calculate all the means is 91. 


Turnitin is a third-party (paid) plug-in that enhances the core Moodle system. Its digital plagiarism- 
detection utility that gives teachers the ability to grade and check papers for plagiarism. Because plagiarism 
is a growing concern in on-line education, Turnitin is an effective way for instructors to check for it 
(Sutherland- Smith & Carr, 2005). The system compares student work to past papers, articles, and books 
from internal sources as well as other university systems and provides color-coded originality scores and 
reports for use by instructors (Dahl, 2007). The system also has many additional features; for example, 
Turnitin provides digital feedback, an online grade book, and attendance lists. 

3.1.2 Models Analyzed 

The first model uses ease of assessment tools (M Assess) in Moodle as the dependent variable. An OLS 
regression analysis was performed to report the results. The “i” subscript attached to each variable refers to 
the fact that response rates were obtained from separate students. 

M Assessi = Constant + ai M_Accessi + <X2 MComrm + a3M_Expi 

+ 04 Traim + as Genden + 06 Standingi + a 7 GPAi + as Collegei + Sit ( 1 ) 

The second model uses the “Overall Satisfaction” experience of each student with using Moodle as the 
dependent variable. Since M Assess and OverallSatisfaction of students with Moodle are jointly 
determined we use two-staged least square (2SLS) analysis to test the incremental effect of M_Assess on 
Overall Satisfaction. 2SLS analysis of Overall Satisfaction as the dependent variable and M Assess as the 
instrumental variable controls for the endogenous relationship between them. 

Overall_Satisfactionit = Constant + Pi M_Assessi + P 2 M_Accessi + PsM Comirii 

+ P 4 MExpi + Ps Traim + P6 Genden + P 7 Standingi + Ps GPAi + p? Collegei + Sit (2) 


4. RESULTS 

Table 2 descriptive statistics indicate that most of our respondents were female sophomore students taking 
summer classes offered by the College of Liberal Arts and Sciences or the College of Business. The students 
typically took two or three courses over two summer sessions, and most of their courses used Moodle, which 
they valued (i.e., felt it was important) in their classes. In terms of how students used Moodle, 68.2% felt it 
was straightforward and intuitive, 75 . 9 % found it was easy to navigate, and 85.8% reflected that it enabled 
them to easily complete class assignments. Overall, only 11.3% were dissatisfied after using Moodle for 
these courses (70.8% satisfied and 18% neutral). It is worth noting that even though the majority of students’ 
comments indicated that Moodle was easy to use, only 17.6% of the students received Moodle training. 
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In comparing the subsection of 72 students whom did not receive an orientation to Moodle, the findings 
were consistent with the findings from those who did receive Moodle training. 

Table 2 shows that the mean score for “Overall Satisfaction” is 2.2 (where a score of 1 is very satisfied, 2 
is satisfied, and 3 is neutral), and the majority of students felt it was easy or very easy to view grades and 
access assignment feedback in Moodle. Nevertheless, only 43% of students felt it’s easy or very easy to use 
when the written assignments were submitted via the Turnitin functionality. The range of this response 
distribution resulted in an average score of 3.5 out of 5 for the M_Assess variable. 

The range of response rates for questions, which measured the ease of accessing (i.e., M Access) the 
various communication and collaboration functions in Moodle, varied from a low of approximately 52% for 
viewing Panopto (lecture capture) recordings to a high of 86% for completing assignments using Moodle. 
This resulted in an average score of approximately 4.0 out of 5.0 for the M Access variable. Our final 
explanatory variable is labeled M Comm, and it measured the ease of use of the common 
communication/collaboration tools in Moodle. The M_Comm, or the ease of communication measure, 
ranged from a low of approximately 17% for using the blog feature to a high of 53% for sending messages 
via e-mail. This probably reflects the student expectation that, with online courses, instructors will use 
numerous advanced access features (e.g., forums, blogs, and chats) rather than simply communicating via e- 
mails. Hence, this resulted in the lowest score of 3.3 for the M_Comm variable. The highest score of 3.9 out 
of 5.0 was for the MExp measure, which is attributable to the students’ experience with Moodle. 


Table 3. Person Correlation Coefficients 


Variable 

Train 

M Assess 

M Access 

MComm 

MExp 

Gender 

Standing 

GPA 

College 

Overall 

Satisfaction 

0.02 

-0.65*** 

-0.74*** 

-0.60*** 

-0.84*** 

-0.01 

0.16 

-0.0 

-0.16 

Train 


-0.04 

0.13 

0.11 

0.03 

0.07 

-0.06 

-0.10 

0.01 

M Assess 



0.64*** 

0 jg*** 

0.62*** 

-0.12 

-0.14 

0.03 

-0.05 

MAcess 




0.76*** 

0.82*** 

0.10 

-0.31** 

0.03 

0.14 

MComm 





0.72*** 

0.16 

-0.44** 

-0.01 

-0.16 

MExp 






0.05 

-0.20 

0.07 

0.14 

Gender 







-0.16 

0.29** 

0.20 

Standing 








-0.16 

-0.08 


***. Correlation is significant at the 0.01 level (2-tailed); **. Correlation is significant at the 0.05 level (2-tailed) 


Table 3 shows that students’ overall satisfaction with using Moodle is significant and negatively 
correlated at the p < 0.01 level with all our explanatory variables M_Assess, M_Access, M_Comm, and 
M Exp. A lower score for the overall satisfaction dependent variable refers to greater satisfaction, and a 
higher score for the explanatory variables refers to greater ease of use of using the functionality in Moodle. 
Thus, the significant negative relationship suggests that the higher the score for the explanatory variable, the 
greater the level of satisfaction with Moodle. Table 3 also shows that the student’s class standing is an 
important control variable. Our summer course data suggests that freshmen and sophomore students are 
more comfortable with the assessment and access functionalities of Moodle as compared to juniors, seniors, 
and graduate students; this is possibly explained by older students' having to “unlearn” the previous campus 
LMS. 

Table 4 shows the OLS regression analyses with M_Assess as the dependent variable. It shows that the 
M_Assess variable is significantly and positively associated with the M_Comm, and that our model 1 
explains 57% of the variance in M_Assess that is significant at p < 0.01 level. This result suggests that using 
Moodle to do online assessment activities, such as providing feedback on assignments, is significant and 
positively influenced by the extent to which instructors use the communication and collaboration tools. 
Thus, we conclude that there is significant support for HI that using Moodle for assessment activities is 
significantly and positively associated with the instructors’ familiarity with using the tools, particularly the 
advanced functionalities offered by the LMS. Further, the table also shows that gender, particularly females 
are more critical than males of learning assessments in Moodle. 
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Table 4. Regression of Assessment in Moodle (M Assess) on Explanatory Variables and Control Variables 


Variable 

Beta 

Std. Error 

t 

Sig. 

Constant 

-0.10 

1.91 

-0.05 

0.96 

M Access 

0.05 

0.35 

0.17 

0.87 

M Comm 

0.53 

0.29 

2.21 

0.04** 

M Exp 

0.31 

0.38 

1.13 

0.27 

Train 

0.06 

0.43 

0.42 

0.68 

Gender 

-0.25 

0.39 

-1.69 

0.10* 

Standing 

-0.06 

0.20 

-0.40 

0.69 

GPA 

0.07 

0.22 

0.48 

0.64 

College 

-0.01 

0.10 

-0.01 

0.99 


Adjusted R 2 = 0.57, Model’s F stat 5.31 that is significant at the 0.002 level; **. Correlation is significant at the 0.05 


level. 

Table 5 shows that the Overall Satisfaction with using the Moodle LMS is significantly and negatively 
associated with the M_Exp variable, and that our model 2 explains 60% of the variance in the level of student 
satisfaction that is significant at the p < 0.01 level. This result suggests that the higher the positive 
experience with navigating the system, completing assignments, and interaction to do things in Moodle, the 
greater the overall student satisfaction. Thus, we conclude that there is significant support for the hypothesis 
that satisfaction with using Moodle for online learning is significantly influenced by the instructors’ 
familiarity and knowledge about how best to structure the navigation of the courses in Moodle as well as the 
use of it to complete assignments. 


Table 5. 2SLS Regression of Overall Satisfaction with Moodle on Explanatory and Control Variables 


Variable 

Beta 

Std. Error 

t 

Sig. 

Constant 

5.67 

1.84 

3.08 

0.01*** 

M Assess 

-0.03 

0.23 

-0.11 

0.91 

M Access 

0.03 

0.34 

0.12 

0.91 

MComm 

0.03 

0.32 

0.09 

0.93 

M Exp 

-0.90 

0.38 

-3.30 

0.00*** 

Train 

0.05 

0.41 

0.34 

0.74 

Gender 

0.38 

0.41 

0.93 

0.37 

Standing 

0.04 

0.19 

0.30 

0.77 

GPA 

0.05 

0.22 

0.35 

0.73 

College 

-0.05 

0.10 

-0.38 

0.71 


Adjusted R 2 = 0.60, Model’s F stat 5.38 that is significant at 0.001 level. 


5. CONCLUSION 

In comparing students’ perceptions with that of the 17 faculty members, we found that faculty members felt 
the Moodle interface was straightforward (50%), easy to navigate (80%), and satisfactory to create class 
assignments (81%). Of the faculty members, 82% had been trained to use the LMS. Additionally, no one 
was dissatisfied after using Moodle during summer 2013, which was the first time that online courses were 
offered by this university. In terms of the more assessment-based and interactive/collaborative activities 
within Moodle, faculty reported less frequent use than did students of Moodle assignments (63%), Turnitin 
assignments (35%), graded discussion forums (35%), messages (35%), wikis (34%), quizzes (29%), chats 
(29%), and blogs (6%). However, 100% of instructors were 1) comfortable with Moodle, 2) able to add their 
syllabi and make their courses available, and 3) able to easily upload files and content to their sites. 
Additionally, 94% felt they were able to communicate effectively with their students using Moodle. These 
four areas were focal points for the university when providing professional development to the faculty. As a 
result, instructors were less likely to use more advanced functions of the LMS (e.g., Turnitin assignments, 
graded forum discussions, quizzes, and chats), beyond those basic functions in which they received training. 
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Hence, the faculty finding further supports our hypothesis that using Moodle for assessment activities is 
significantly and positively associated with the instructors’ familiarity with using such tools, particularly the 
advanced functionalities offered by the system. Furthermore, there is support for the hypothesis that 
satisfaction with using Moodle for online learning is significantly influenced by the instructors’ familiarity 
and knowledge about how best to structure the navigation of the courses in Moodle and use it to complete 
assignments. 
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ABSTRACT 

The study examined the interaction between field dependence-independence (FD/I) and learning with modeling software 
and simulations, and their effect on children’s performance. Participants were randomly assigned into two groups. Group 
A first learned with a modeling tool and then with simulations. Group B learned first with simulations and then with a 
modeling tool. A statistically significant interaction was found between FD/I and the order of using the two types of 
software. FI children in group A outperformed FD children in the same group on the modeling task. However, these 
results were not observed with the FD children in group B indicating that learning first with simulations facilitated the 
subsequent learning with the modeling tool of FD children only. 


KEYWORDS 

Modeling tools, simulations, field-dependence/independence, interaction effect, problem solving 


1. INTRODUCTION 

The use of computers in pre-school education is an issue of great concern for the research community (Chen 
& Chang, 2006). The objective and aim of every relevant research is to understand the way in which 
computers should be used, so as to facilitate the learning of young children (Haugland. 2000). In 2012, the 
internationally acclaimed organization National Association for the Education of Young Children (NAEYC), 
having as its main concern the overall development of children, and more specifically of infants, published 
guidelines according to which pre-primary teachers are expected to use computers in the classroom. One of 
these guidelines states that the use of computers should only be done when used in a correct pedagogical way 
aiming at the children's overall development. Since children can use computers with ease (Clements, 1999), 
the use of such tools in organized learning environments (e.g., in classrooms) is considered necessary in 
order to encourage students to investigate, create, solve problems and think critically. To maximize, however, 
the benefits for the children, computers have to become an integral part of teaching and learning throughout 
the curriculum (Chen & Chang, 2006) and always within an organized framework where computers are used 
in a developmentally suitable manner aiming at the children's overall development. 

In particular, the use of modeling and simulation tools in young children’s education is a research field of 
great interest (Dimitracopoulou, Komis, Apostolopoulos, & Politis, 1999; Jacobson & Wilensky, 2006; 
Sterman, 2006; Hmelo, Holton, & Kolodner, 2000; Ayersman, 1995; Moreno & Meyer, 1999). Modeling 
tools are powerful mental tools for the creation and understanding of systems through a process of studying 
the relations among the variables of the entities comprising the system (Jonassen, 2000, 2004). Simulation 
tools constitute software dynamically visualizing the relations among entity variables (Clariana & Strobel, 
2008). As Rieber (1996) states, simulations are governed by the environment determined mainly by the 
content. Their design includes the mental models of their creators (e.g., teachers), and the students have to 
handle the variables and observe the relations between them (Clariana, 1989). As Linn et.al (2010) mention, 
interactive simulations as intermediate models facilitate the learning of abstract phenomena in authentic 
contexts and the understanding of the relations between variables. The modeling software represents a 
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complex phenomenon showing the quantitative or qualitative relations between variables, without however 
offering the rich authentic dynamic representations of the simulation software. 

Another factor influencing the understanding of the content when taught with the use of any technological 
tool is directly related to the cognitive characteristics of the students themselves (Hmelo & Azevedo, 2006). 
For example. Dragon (2009) argues that individuals with a different cognitive style process data and 
information in a different way. In Educational Technology, the cognitive style of field- 
dependence/independence (FD/I) has been quite popular among the researchers. The term cognitive style 
refers to individuals’ different abilities to absorb and process information from their environment (Burnett, 
2010; Chen & Macredie, 2002; Morgan, 1997; Witkin, Moore, Goodenough, & Cox, 1977). FD/I is 
characterized as a dipole, with the one end including individuals whose characteristics refer to the field- 
independent (FI) cognitive style, and the other end including individuals whose characteristics refer to the 
field-dependent (FD) cognitive style (Morgan, 1997). The main difference between the two ends, according 
to researchers, is found in the way that individuals process complex representations (Morgan, 1997; Chen & 
Macredie, 2002; Davis, 1991; Snowman & Biehler, 1993; Canelos, Taylor, & Gates, 1980; Liu & Reed, 
1994). 

Although there is insufficient research in the literature regarding the examination of the role of the 
cognitive style on young children’s cognitive performance during their learning using simulations and or 
modeling software, there is recent research evidence showing the effects of FD/I on undergraduate students' 
ability to solve a complicated problem using modeling tools (Angeli & Valanides, 2004; Angeli, Valanides, 
& Kirschner, 2009; Angeli, 2013; Angeli & Valanides, 2013) and simulation software (Trees, Doyle, & 
Radzicki, 1996). Specifically, the research evidence thus far shows that FI learners outperform FD learners 
during problem solving with modeling tools. Within the pre-primary education context, there is only one 
study conducted by Polemitou (2013), which also showed that FI children outperformed FD children when 
they worked with a modeling tool. 

In the early 1990s, there were many concerns, which even led to a debate to some extent, about the order 
in which types of software should be integrated in the education of young children. More specifically, the 
debate related to the order of use of modeling and simulation software; that is to say if children of a young 
age should learn using modeling software first followed by simulation software or the other way around 
(Nichol. 1988; Webb, 1994; Bliss et al., 1992; Bliss, 1994; Polemitou, 2013). There has been no conclusion 
to this dispute, because, up until today, the research community has not examined this issue systematically 
through empirical research. 

Therefore, to remedy for this lack of research, the study herein examined whether the type of software 
(simulation/modeling) and FD/I affected the performance of children aged between 5 and 6.5 years, as well 
as whether there was any interaction between the order of use of the software (that is to say, the use of 
modeling software first followed by simulation software or the other way around) and children’s cognitive 
style. 


2. METHOD 

2.1 Participants 

Children between 5 to 6.5 years of age participated in the research. The Children's Embedded Figures Test 
(CEFT; Karp & Konstadt, 1971) was used to measure children’s cognitive style. The CEFT test was used 
with 140 children who were recruited for participating in this research study. Only 59 out of the 140 children 
managed to achieve the required grade on the CEFT in order for the researchers to be able to classify them in 
one of the two cognitive styles (i.e., FD or FI). The remaining 81 children were excluded from the research. 

2.2 Data Collection Instruments 

2.2.1 Pretest/Posttest: The Life Cycle of Butterflies 

The test consisted of four exercises. In the first exercise, the children were asked to match pictures of a 
caterpillar, a butterfly, eggs, and a chrysalis with the correct words. In the second exercise, they were asked 
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to put the life stages of a butterfly in order. In the third exercise, the children were given four life cycles of a 
butterfly and four weather conditions, namely, sunny, cloudy, rainy, and snowy, and they were asked to 
match each life cycle with a weather condition. In the fourth and last exercise four levels of sunlight were 
presented. The children had to circle the chrysalis or the butterfly for each level of sunlight, depending on 
what they believed would happen, that is to say if the chrysalis would turn into a butterfly or whether it 
would remain a chrysalis, based on the given level of sunlight. 

2.2.2 Pretest/Posttest: The Life Cycle of Bees 

The test consisted of five exercises. The first exercise included pictures of three bee types (e.g., worker, 
queen, drone bees), and asked the children to match them with the correct words. Subsequently, in the 
second, third, fourth and fifth exercises they were given pictures of various weather conditions (e.g., sunny, 
cloudy, rainy, snowy), as well as pictures of the tasks performed by bees; the children had to circle the task 
they believed would take place under each weather condition. 

2.2.3 Children's Embedded Figures Test (CEFT) 

The CEFT test (Karp & Konstadt, 1971) is specifically designed for children between 5-9 years of age. Karp 
and Konstadt (1971) reported that the internal reliability of the test, which was measured using Cronbach's 
alpha, was 0.87. The test consists of 38 questions. In the first part of the test the children are asked to locate a 
simple shape in the form of an equilateral triangle embedded in 19 other complex shapes. In the next 19 
questions, the children are asked to locate a small house. The time the children have at their disposal to locate 
the simple shape embedded in each one of the complex shapes is 30 seconds. 

2.3 Modeling Software: “The Life Cycle of Butterflies” 

The software was designed to represent the life cycle of butterflies in the form of a model, and included the 
main factors influencing their survival and the continuation of the species. The child selects one of the four 
weather conditions (i.e., sunny, cloudy, rainy, snowy) first, and. then, the software models the life cycle of a 
butterfly according to the weather condition selected by the child. The variables of the model constitute: (a) 
the quantity of eggs laid, (b) the existence of adequate food quantity for the caterpillar, (c) the size of the 
caterpillar after four days of continuous food consumption, and (d) the quantity of sunlight required for the 
final metamorphosis of the caterpillar from chrysalis to butterfly. The child eventually has to select all four 
weather conditions in order to understand the factors influencing the growth of a butterfly under each 
condition. 

2.4 Simulation Software: “The Life of Bees and their Role in Honey 
Production” 

Using the simulation software, the child selects one of four weather conditions (i.e., sunny, cloudy, rainy, 
snowy), first, and. then, the software simulates the life of bees inside and outside the hive through a series of 
dynamic visualizations. In this way, the children observe the bees coming out of the hive, gathering pollen 
and returning, as well as the mating process between the queen and the drones. They can also observe, 
through a series of animated images, the different tasks being carried out in the hive (i.e., placing nectar in 
the cells, placing wax in the cells, cell purification, and larvae birth). Gradually, the children have to check 
all four weather conditions, in order to understand how weather conditions affect the work and lifestyle of 
bees in different ways. 

2.5 Research Procedure 

During the first stage, each child was given the CEFT test individually (Karp & Konstadt, 1971). Based on 
the children’s performance on this test, the researchers classified them in one of the two cognitive styles, 
namely, FI or FD. The total duration of the CEFT test was 20 minutes. Each cognitive style classification was 
then randomly divided into two sub-groups; group A worked with the modeling software first followed by 
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the simulation software, and group B worked with the simulation software first followed by the modeling 
software. 

Fifteen days after the first stage, the second stage followed. In the second stage, the children of group A 
were given a pretest to assess their prior knowledge regarding the life cycle of butterflies, while the children 
of group B were given a pretest about the life cycle of bees. The assigned time for each pretest was 10 
minutes. After the children completed the pretest, they were asked to work on the computer for the next 30 
minutes, using the modeling software (Group A) or the simulation software (Group B). 

During the first 10 minutes of the third stage, the posttest was administered. Then, the children had a 10- 
minute break, and those who worked with the modeling software in the second stage were given a 10-minute 
pretest to assess their prior knowledge regarding the life of bees, while the children who worked with the 
simulation software in the second stage took a 10-minute pretest to assess their prior knowledge regarding the 
life cycle of butterflies. After the tests were completed, the children were asked to work on the computer for 
the next 30 minutes, using either the simulation or the modeling software. Children who in the second phase 
worked with the modeling tool they were asked to work with the simulation software, and those who worked 
with the simulation software in the second phase worked with the modeling tool during the third phase. The 
posttests were administered two days after the third stage. 


3. RESULTS 

Table 1 presents descriptive statistics of FD and FI children’s performance on the pretests. 


Table 1. Descriptive statistics of FD and FI children’s performance on the pretests 


Modeling pretest 

Simulation pretest 



Mean 

SD 

N 

Mean 

SD 

N 

FD 

65.75 

19.57 

34 

52.94 

12.76 

34 

FI 

75.42 

15.82 

25 

57.26 

14.36 

25 


The performance of FD children on the modeling pretest was not particularly high (M = 65.75, SD = 
19.57). Similarly, their performance on the simulation pretest was also low (M = 52.94, SD = 12.76). FI 
children had a higher performance, M = 75.42, SD = 15.82 and M = 57.26, SD = 14.36, on the modeling and 
simulation pretests, respectively. Nevertheless, while FI children performed better on both pretests, the 
difference between FI children’s performance on the simulation pretest and FD children’s performance on the 
same test was small. 

Table 2 presents descriptive statistics of FD and FI children’s performance on the posttests. 

Table 2. Descriptive statistics of FD and FI children’s performance on the posttests 




Modeling posttest 



Simulation posttest 



Mean 

SD 

N 

Mean 

SD 

N 

FD 

93.93 

7.91 

34 

68.42 

16.89 

34 

FI 

98.00 

5.91 

25 

86.31 

13.93 

25 


The FD children’s performance on the modeling posttest (M = 93.93, SD = 7.91) was higher compared to 
their performance on the pretest (M = 65.75, SD = 19.57). The same was observed with the performance of 
FI children (M = 98.00, SD = 5.91). Flowever, FI children’s scores on the modeling posttest were higher than 
the scores of the FD children. Regarding the simulation posttest, scores of both the FD and FI children, M = 
68.42, M = 86.31, respectively, were much higher than their pretest scores. 

Table 3 presents descriptive statistics on children’s performance on the modeling and simulation posttests 
for each cognitive style (FI/FD) and the order of use of the software. 
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Table 3. Descriptive statistics of children’s performance on the modeling and simulation posttests per cognitive style and 

group 


Modeling posttest Simulation posttest 



Mean 

SD 

N 

Mean 

SD 

N 

FD 

Group A 

90.97 

8.89 

18 

86.55 

15.76 

18 

Group B 

97.26 

5.08 

16 

88.49 

18.55 

16 

FI 

Group A 

99.10 

3.34 

14 

87.59 

10.03 

14 

Group B 

96.59 

8.08 

11 

84.68 

18.15 

11 


Notes: Group A = modeling-simulation. Group B = simulation-modeling 


As evidenced by the descriptive statistics above, the FI children of group A, namely, those who learned 
using modeling software first followed by simulation software, had a higher average performance than the 
FD children of group A on both the modeling and simulation posttests. On the other hand, FD children in 
group B, namely those who learned using simulation software first followed by modeling software, 
performed better than the FI children in group B on both posttests. Furthermore, the average performance of 
FD children in group B (M = 97.26, SD = 5.08) had a better performance than the FD children in group A on 
the modeling posttest (M = 90.97, SD = 8.89), and better performance than FI children in group B on the 
same test (M = 96.59, SD = 8.08). 

To examine whether the order of use of the software affects FD and FI children's performance in a 
different way, taking into consideration their prior knowledge, a 2X2 Multivariate analysis of covariance 
(MANCOVA) was carried out. The results showed that the children’s performance on the modeling pretest 
did not affect their performance on the modeling (F (1 53) = .43, p= .51 ) and simulation (F ( i 53) = 1.56, p = .21) 
posttests to a statistically significant extent. The performance of children on the simulation pretest affected 
their performance to a statistically significant extent only on the simulation posttest ( F ( 1 . 53 )= 7.23, p <.05 ), 
but not their performance on the modeling posttest (F ( 153 )= .00, p = .95). The results also showed a 
statistically significant interaction between the cognitive style and the order of use of the software, regarding 
the performance of students on the modeling posttest (F (U3) = 5.17 , p < .05). Specifically, FI children in 
group A outperformed FD children in group A on the modeling task, however, these results were not 
observed with the FD children in group B indicating that learning first with simulations facilitated the 
subsequent learning with the modeling tool of FD children only. 


4. DISCUSSION AND SIGNIFICANCE 

According to the results, there is a statistically significant interaction between cognitive style and the order of 
using the two types of software. More specifically, FI children, who first used the modeling tool and then the 
simulation software, had a statistically higher performance on the modeling posttest, than FD children, who 
also used the computer tools in the same order. These results, however, were not observed when FD children 
first used the simulation software and then the modeling tool, which means that, learning using simulation 
software facilitated the subsequent learning with modeling software of FD children only. The results of this 
study are significant, because they show that FD learners can also learn about complex systems with 
modeling software if they learn first with simulation software and then with modeling software. This is 
highly important and significant for both the research community, as it informs the long time debate 
regarding the order of using the two types of software, but also the classroom teacher in terms of deciding 
about what tool (simulation of modeling) to integrate first in the education of young children. 
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ABSTRACT 

There have been many reports on cases where flipped classroom was applied which put greater emphasis on conducting 
various learning activities during class. However, there is a limitation in redesigning existing university lectures as 
flipped classrooms merely based on reports that describe the learning activities of and their effects on individual 
subjects. Therefore, for this study, flipped classroom was applied to and managed two different courses and a comparison 
was made on different application types and learners’ responses depending on the different instructors, learners, subjects 
and teaching methods of the class before the redesign. The result showed that the types and numbers of learning activities 
redesigned into flipped classrooms turned out to be different depending on the characteristics of the learners, the teaching 
methods of the previous class, and the types of activities that instructors thought were significant but insufficient. 
Moreover, surveys and interviews were conducted on positive and negative aspects of online video, learning activities, 
homework, and the overall flipped classrooms. We suggest different opinions according to characteristics of learners and 
composition of learning activities as a result. This study has its significance in demonstrating that various kinds of 
flipped classrooms can be designed and managed in accordance with various compositions of different courses while 
providing an overall guideline on how to design the course based on the different factors of each course. 


KEYWORDS 

Flipped Classroom, Instructional Design, Case Study 


1. INTRODUCTION 

Lecture-centered education is one of the most commonly used methods as it can deliver a great amount of 
knowledge within a short period of time. However, it has also been heavily criticized for being unable to 
nurture higher -level thinking such as critical, problem-solving, and creative thinking. Moreover, it is also ill- 
suited to developing the social knowledge capacity that is one of the important goals of education and 
facilitated through interaction between instructors and co-learners. 

To improve higher-level thinking and social knowledge capacity through communication and interaction, 
various learner-centered education methods such as Problem-Based Learning, Situated Learning, and Goal- 
Based Scenarios have been suggested as an alternative to one-sided knowledge delivery. Based on such 
changes in the education paradigm, flipped classrooms have recently gained particular interest among experts. 
In flipped classrooms, learners listen to the lecture in advance through online video before class; and for the 
parts they have failed to fully understand through the online video, they actively engage in problem-solving 
activities either through discussions with co-learners or through the help of teaching assistants and instructors 
during the class (Bates & Galloway, 2012). Emphasizing the concept of ‘flipped (inverted),’ Bergman and 
Sams (2012) defined flipped learning as ‘the type of learning in which activities traditionally conducted in 
classrooms are done at home and conversely, activities usually done at home are conducted in classrooms. ’ 

Examples of university courses where inverted learning was applied are the Software Engineering course 
at Miami University (Gannod, Berge, & Helmick, 2008), the Basic Information Engineering course at 
Bentley University (Frydenberg, 2013), the Genetics and General Biology course at University of Missouri 
(Stone, 2012), and the Calculus course at University of Michigan (Berrett, 2012). In Korea, KAIST and 
UNIST have been running flipped classrooms since 2012. In particular, KAIST started to run three courses 
implementing flipped classroom in the spring semester of 2012 and expanded it into 10 courses during the 
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fall semester of 2012; throughout the spring and fall semesters of 2013, a total of 62 courses in KAIST were 
run in flipped classrooms and the number is expected to further increase (Lee, 2012). 

Case studies of these universities where inverted classrooms were applied provide data on management 
formats, composition factors and effects of the flipped classrooms. In most cases, flipped classrooms were 
applied to subjects in the field of science and engineering and were generally composed of online video 
lecture before class, quizzes, discussion, problem-solving and feedback during class, and wrap-up and review 
after class. There have been significantly meaningful effects in terms of learners’ achievement, satisfaction, 
and participation in flipped classrooms (Bates & Galloway, 2012; Frydenberg, 2012; Gannod, Burge, & 
Helmick, 2008; Johnson & Renner, 2012). However, there are limitations in effectively redesigning existing 
university lectures into flipped classrooms based solely on description of individual subject’s learning 
activities and their effects. Even though the classes deal with the same learning content, they need to be 
designed and applied to different types of flipped classrooms in accordance with the characteristics of 
subjects, learners, and instructors as well as the teaching method of the previous class. 

This study applied and ran inverted learning in two different courses; and based on the results, it 
examined different application types and learners’ responses to inverted learning depending on instructors, 
learners, subjects and the teaching method of the previous class and learners’ responses to them. 


2. METHODS 

For this study, the Calculus 2 and Nonlinear System Theory courses opened at Seoul National University in 
Korea were redesigned into flipped classrooms during the 2013 fall semester. For each course, one person 
with a doctoral degree in educational technology and one graduate student supported the instructional design 
and management; and instructors and teaching assistants were closely consulted for course application. 
During every class, observation was made through either manned or unmanned filming and surveys were 
conducted of every student during mid-term and final examinations. After the end of the semester, five 
learners were closely interviewed. 

The survey conducted during the mid-term exam aimed to gauge the general response of the class and get 
feedback from the students for future classes; survey conducted during the final exam aimed to find out the 
effects of the class methods applied to each course and the learners’ responses. This study made use of some 
parts of the mid-term survey items, which enable comparison of the two courses. Following is a table on the 
different characteristics of the two courses based on the syllabi, instructor interviews and learner surveys. 


Table 1. Characteristics of courses where flipped classroom was applied 




Calculus 2 

Nonlinear System Theory 

Subject 

Categories 

Compulsory basic liberal arts 
course 

Elective course within the major 

Level 

Mid-Low 

High 

Prerequisite subjects 

Calculus 1 

Linear system theory 

Language 

Korean 

English 

Video materials 

- 

Class for 2011 academic year 

Learner 

Grade/Number 

Freshmen among 
undergraduates/ 1 8 

Master's and doctorate course at graduate 
school/ 12 

Composition of 
departments 

Mathematics Education 

Four departments, apart from Dept, of 
Electrical and Computer Engineering 

Prerequisite subjects 

Completed by all learners 

Completed by nine out of twelve learners 

Instructo 

Experience with 
running the course 

Ran for the first time in a decade 
(ran by professors of the 
Department in rotation) 

Have run for the last decade 

Previous teaching 
method 

lecture + problem-solving through 
team discussions 

Lecture focusing on concepts by using 
PPT 


Motivation for 
applying flipped 
classroom 

Improving understanding on 
conceptual equations and formulas 
through various visual materials 

Enhancing understanding of subject 
matters through English online video 
lecture before class + class in Korean 
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Igniting learners' motivation by 
implementing more diverse and 
larger learning activities 

Identifying and correcting learners’ 
misunderstanding through problem- 
solving 

Inducing questions from learners 


Aside from the fact that the two classes included content related to mathematics and were small-size 
classes with less than 20 learners, the two courses had totally different characteristics, as suggested by the 
table above. 


3. RESULTS 

3.1 Composition of Classes Designed for Flipped Classroom 

As the two courses were originally different in terms of subjects, learners, and instructors, they were 
naturally redesigned and managed as different types of flipped classrooms. Following is a table which 
provides comparisons on the learning activities between the previous class and the flipped classroom. 


Table 2. Comparison of the composition of learning activities between the previous class and the flipped classroom 




Calculus 2 

Nonlinear System Theory 


Before 

applying 

flipped 

classroom 

After 

applying 

flipped 

classroom 

Before 

applying 

flipped 

classroom 

After applying flipped classroom 

Before Clas 
(Online) 

Video 

learning 

- 

O 

A 

O 

Questions- 

Answers 

- 

- 

- 

A 

In Class 
(Online) 

Quiz 

- 

15% 

- 

- 

Lecture 

70% 

15% 

80% 

20% 

Problem 

solving 

Team 

discussion- 

presentation 

30% 

70% 

- 

- 

40% 

Questions- 

Answers 

A 

A 

A 

20% 

Instructor- 

led 

problem- 

solving 

- 

- 

20% 

60% 

20% 

After Class 
(Online) 

Assignment 

Reflection 

Journal 

1 per class 

1 per class 

- 

- 

RSR 

- 

4 times 

- 

- 

Reports 

- 

1 time 

- 

- 

Questions- 

Answers 

- 

- 

- 

A 


In the Calculus 2 course, online video learning which was not included in the previous class and quizzes 
to check whether learners had done their learning or not were added; Resource Search & Review (RSR) and 
reports were added as part of homework while the proportion of problem-solving activities through 
learner-centered discussion and presentations was increased. Compared to the previous class, the proportion 
of learning activities learners in the flipped classroom had to do before and after class rose. In the Nonlinear 
System Theory course, at the beginning of the semester, only voluntary Q&A was added before and after 
class to lead the class mostly through questions and answers. However, after the mid-term, team discussions 
and presentations were added to encourage class interaction. It can be said that instructor -led problem- 
solving was the major activity before the mid-term exam while new activities such as team discussions and 
presentations emerged as the major activities after the mid-term exam. 
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3.2 Composition of Classes Designed for Flipped Classroom 


Following are the learners’ responses to the inverse learning based on the survey of positive and negative 
aspects of prior online video learning, learning activities during class, after-class homework, and the overall 
inverse learning, which can be found in both the Calculus 2 and Nonlinear System Theory courses. 


3.2.1 Learners’ Responses to the Prior Online Video Learning 

Learners in both courses responded positively that repetitive learning was possible through online videos. 
Such positive responses prevailed among learners as they could use them before class and for preparing for 
exams. It was followed by the response that they felt burdened under an obligation to engage in mandatory 
study separately from the official class. Some also gave a negative response that online video lecture were 
not that efficient compared to classroom lectures. Other responses included that learners were able to adjust 
the learning time and place for their convenience (Calculus 2) and it was challenging to learn through online 
video lectures delivered in English (Nonlinear System Theory). Following is the table that shows learners’ 
responses to the online video learning before class. 


Table 3. Learners’ responses to online video learning before class 


Learners’ response (Multiple Response) Calculus 2 

Pros/Con Response 

s 

+ It is helpful because 1 can repeatedly listen to the parts that 

1 do not fully understand. 

It is burdensome as I am under an obligation to study in 
addition to the class. 

Online video lectures are less efficient than lectures ^ 

delivered during class. 

+ It was good to be able to adjust learning time and place at 

my convenience. 

+ It was good since I could skip the parts that I know and 

selectively listen to the parts that I find it difficult. 

I hardly ended up listening to the video lecture as it was 
provided through online. 

It was regrettable to have less interaction that usually 
occurs in the classroom. 

Even though online lectures can be taken repeatedly, it 
was burdensome to take lectures delivered in English 
through online. 

Once one gets behinds the schedule, one helplessly falls 
behind the whole course. 


Nonlinear 

System 

Theory 


9 

5 

3 


2 


1 

1 


3.2.2 Learners’ Responses to Learning Activities during Class 

Learning activities for the Calculus 2 course were composed of quizzes and cooperative learning. The largest 
number of respondents said the quizzes were not helpful to learning (8 learners), but at the same time a large 
number of people also responded that learning was naturally induced as they were checked whether they 
viewed the online video lecture or not (8). Learners probably felt the quizzes were relatively challenging as 
they got to know the learning content for the first time during the quiz time, and also burdensome as the quiz 
scores were reflected in course credit. Learners gave positive reviews for the collaborative learning activities, 
which proved to be helpful to problem-solving as they were able to engage in various kinds of thinking (5), 
enhance their understanding (4) and exchange ideas with each other (3). Other opinions included 
‘cooperative learning itself was very interesting,’ and ‘the more challenging the problem is, the more 
efficient it will prove to be. ’ 

Learners enrolled in the Nonlinear System Theory course were required to view the online video lectures 
and raise any questions either through LMS before class or directly during class. In addition, learning 
activities were mostly focused on questions and answers during class. First, learners were asked what kinds 
of answering methods they preferred during the Q&A time at the class. The result showed that they preferred 
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the most when one or several learners answered the questions while the instructor added relevant opinions 
and additional answers (5). This was followed by insUuctor-led Q&A type (4). In conclusion, learners were 
overall satisfied with the Q&A activities during their inverted learning; and new measures were drawn up to 
expand learners’ participation and have some time for brief class review of the online video at the beginning 
of the class. 

3.2.3 Learners’ Responses to After-Class Homework 

Daily reviews, RSR (Resource Search & Review) and team reports were given to the learners as homework 
for the Calculus 2 course. Learners’ opinions on individual homework such as daily reviews and RSR were 
that daily reviews were helpful (6) for reviewing the learning content (5) and enhancing understanding (2). 
However, some responded that daily reviews were not helpful at all (3). It is probably because they thought 
daily reviews to be inefficient and artificial. Moreover, some pointed out that if questions that genuinely 
encouraged reviews were suggested, it would lead to more productive reviews. 

One of the most frequently suggested opinions in regards to RSR, for which learners searched research 
theses or the Internet such as Navercast and shared such learning with others, was that it did not prove to be 
that helpful (7). It is probably because learners, freshmen in an undergraduate course, are not that familiar 
with researching activities, which led them to feel reluctant and challenged. However, there were some 
positive responses saying that it helped them to increase their interest in mathematics (4), deepen 
understanding (2), and engage in further study (2). Other opinions included improvement of information 
searching ability, time management practice, and unfair reflection in course credit. 

Unlike Calculus 2, the Nonlinear System Theory course gave relatively little homework, which required 
learners to voluntarily upload questions and answers. Completing and submitting homework was not counted 
for credit. Accordingly, after interviewing five learners who took the course, three learners mentioned the 
importance of homework. 

For this subject, it s good to have time for questions, (...) but it would be better and much more efficient if we 

had some problems to solve as homework before class and have some time for discussion over the problems with 
the professor during the class (Interviewee 2). 

As it is a challenging subject, it would have been better if professor gave us some problems as homework 
before class; since we had to solve the problems during class, it was very challenging and even though having 
homework itself maybe burdensome, it might be more efficient if we could think it over before class. (Interviewee 
3). 

It would be more efficient if we had homework. With no homework, we don ( do any problem-solving on our 

own. Regardless of how difficult or easy the course is, homework is necessary. In particular, easy questions 
should be handed out for difficult subjects and difficult questions for easy subjects. That is what homework is all 
about. If too difficult questions are given for difficult subjects, it’s not only impossible but also meaningless. 
Likewise, if too easy questions are given for easy subjects, it really doesn ’t help. (Interviewee 5). 

Based on the opinions provided by learners, it was concluded that Q&A during class could be more 
motivated and efficient if homework and learning materials were provided along with the online video 
lecture learners took before class. In particular, since the course is a graduate school course that is relatively 
difficult and less sensitive with regard to credit, learning materials or homework related to in-class Q&A can 
be suggested during online video lecture prior to the class so that learners have some time to be better- 
prepared for the class. 

3.2.4 Learners’ Responses to the Overall Inverse Learning 

Learners’ overall responses to the flipped classroom can be broadly classified into positive and negative 
aspects. For the positive aspects, learners of both courses gave the same opinions in regards to interaction and 
online video lecture. In terms of interaction, learners held positive views toward flipped classroom as it 
enabled constant and active interaction between insUuctor and learners during class. Moreover, in terms of 
online video lecture, learners positively evaluated it because it made the class more efficient as learners were 
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able to preview the learning before class, participate in the class more easily and repeatedly study the difficult 
parts. 

Learners who took the Calculus 2 course responded that their overall studying time increased and they 
were able to acquire an efficient learning method. Learners who took the Linear System Theory course 
responded that they were able to prepare for in-class Q&A, have in-depth thinking as they reviewed their 
own questions, and prevent themselves from asking questions irrelevant to the class. Furthermore, learners 
responded that they could better understand the lecture through flipped classroom compared to the traditional 
lecture class. Following is a table that summarizes the strong points of flipped classroom pointed out by the 
respondents. 


Table 4. Learners’ Opinions on the Positive Aspects of Flipped Classroom 


Category 

Learners' Opinions 

Calculus 2 

Nonlinear System Theory 

Interaction 

Constant exchanges between instructors and learners and active 
participation 

Video 

Repetitive learning is possible through online video lecture 

before offline class 

Efficient learning and participation 

Learning 

Increased individual learning 
time 

Acquisition of learning 
method 

In-depth thinking over 
learning through Q&A 
Fruitful discussion and 
enhanced understanding 


Negative aspects of flipped classroom pointed out by learners who took the Calculus 2 course were 
focused on too much workload, content limitation, and online video lectures. As it was impossible to follow 
the class without viewing the online video lectures in advance, and plenty of homework was given, learners 
felt heavily burdened; they also complained that the range of subjects covered during flipped class was much 
more limited. Moreover, having no interaction during online video class was also pointed out as another 
negative aspect. However, negative aspects pointed out by learners who took the Linear System theory 
course were mostly focused on the online video lectures. They said that it was very difficult to participate in 
and follow the class without viewing the online video lecture in advance as class was led mostly through 
Q&A between instructors and learners; and they also said it was very boring. Following is the table that 
summarizes the weak points of flipped classroom pointed out by the respondents from each course. 


Table 5. Learners’ Opinions on the Negative Aspects of Flipped Classroom 


Category 

Learners' Opinions 

Calculus 2 

Nonlinear System Theory 

Learning Burden 

Too much homework (daily 
reviews, RSR, etc.) 

Pressure to engage in 
participatory discussion 
Complying with class 
schedule 


Learning Content 

Covering relatively small 
amount of content that one 
should be learning through the 
course 


Video 

No interaction possible 

Impossible to follow the class 
if online video lecture is not 
viewed 
Boring 
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4. CONCLUSION 

After applying flipped classroom to two different courses, this study compared and observed different 
application types and learners’ responses depending on the different instructors, learners, subjects, and 
teaching methods of the original class. The result showed that various types of flipped learning could be 
devised and ran and learners’ responses also differed accordingly. In particular, it was possible to devise and 
manage different types of flipped classroom depending on instructors’ experience and motivation for 
applying flipped classroom; learners’ responses to the overall flipped learning and its relevant activities also 
differed depending on the courses’ category, learners’ purpose in learning and sensitivity to course credit. 
This study is expected to pave the way for further studies on designing and making guidelines for flipped 
classroom based on various factors in different courses. 
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ABSTRACT 

This paper relates the "Agile School", an emerging archetype of the enterprise architecture: "Processes of Organizational 
Leadership" for leading and managing strategies, tactics and operations of forming in Higher Education Institutions. Agile 
School is a system for innovation and deep transformation of University Institutions in which it starts with the identity and 
it ends at the self-governance and social responsibility, around satisfaction and transcendence by the outcomes at 
environment, resources and people. Our work research is based on the "Executables Enterprise Architectures" using "Lean 
and Agile methodologies through of Serious Play paradigm" for linking the senses with emotional and cognitive areas as a 
means to accelerate identification, analysis and problem solving, enhance decision-making and develop commitment 
actions by all stakeholders. With Agile School, it is possible acknowledge and reach consensus to propose architectural 
solutions supported in the holistic views of individuals and collectives, using simulation scenarios of past and future, to 
build concrete and clear architectures that enable innovation and iterative improvement using information and 
communication technologies through of real and virtual environments. With this perspective it could ensure efficient and 
effective quality of design and development according of required products and services. Particularly, Agile School is 
based in the "Educational Process Maturity Model - EPMM/PE" with five high performance levels, in which products or 
services are on a continuous improvement due to innovation, competitiveness, flexibility and productivity. 


KEYWORDS 

Enterprise Architecture, Agile Methods, Simulation, Maturity Models, Quality Assurance. 


1. INTRODUCTION 

Our research on the development of the organization (Llamosa and Mendez, 2010), induce that systems must 
have an Enterprise Architecture: 

• To integrate principles and specifications to influence her performance and development 
organizational [(Dickerson and Mavris, 2010), (The Open Group, 2013), (Tiwana, 2014), and (Zachman, 
2014)]. 

• To structure a compound building blocks for encourage multidisciplinary work and develop and 
evolve, harmoniously, coherently and synchronously, the continent and content on the strategy, the logistics 
and the operation of the organization, following guidelines for efficient and flexible government, to develop 
products and services that are produced in an evolutionary and iterative manner (Bente et all, 2012). 

• To have a set of principles, strategies and tactics to the learning and work collaborative to avoid the 
overhead and bureaucracy, and promote satisfaction on compliance and commitment to delivering value, and 
transcendence on adaptive management, leadership, solve problems and continuing improvement [(Ambler 
and Lines, 2012), (Cardinal, 2014), (Griffiths, 2012), (Schiel, 2010), and Wagener et al, 2011)]. 

This paper gives an account our research realized in the Laboratory of Integrated Organizational Systems 
(LASIO) using “Enterprise Architecture with Agile Methods (ARTEMA)” through developing a pilot of Agile 
School, which extends the "Process Management Model for Higher Education" (Llamosa and Mendez, 2010) 
to two courses with different levels of maturity: Probability and Statistics for Engineers (3th. semester) and 
Quality Engineering (7th. semester) of the program of Electrical and Electronic Engineering. 

This paper is organized as follows: 
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• Organizational leadership processes. This section shows the "what" of the structural, functional and 
operational building blocks, or, enterprise architecture of the administrative management and strategic, 
tactical and operative administration of the "how", "where", "why", "when", "who", "how much" with which 
the feasibility and commitment of value is set. 

• Organizational Maturity Processes. This paragraph describe how assessing compliance of 
organizational leadership processes to monitoring and realizing corrective actions of business continuity and 
preventive actions of research, innovation and development. 

• Agile School Model. This section sets the patterns that serve as reference for organization and 
practices for a typical architecture of Higher Education Institution. 

• Agile School Pilot. This paragraph shows the tailoring that serves as reference for measurement the 
organization and practices achieved by an Institution of Higher Education. This paragraph refers to the pilot 
as architecture type that it has served for the design, verification and validation of Agile School. 

• Conclusions. This fragment recounts the retrospective assessment and future work of the project. 

• Acknowledgment. This part refers the gratitude to those stakeholders contributed to this project. 

• References. In last section include the bibliographic information used in this paper. 


2. ORGANIZATIONAL LEADERSHIP PROCESSES 

Organizational Leadership Processes -OLP- is the first level of our Enterprise Architecture model (Figure 1). 
It consists of an essential element called Administrative Management , which has three administrative building 
blocks: Strategic, Tactical and Operational. Each of the components is observed with holistic common 
perspectives in order to provide comprehensive interpretations of business modeling, and deploying processes, 
followed the perspective of intelligence and decision making, leadership, planning, monitoring and control 
(Figure 2), to develop the tactical administration, which in turn is responsible for providing logistics, manage 
resources, infrastructure, media and knowledge resources, for the operational administration of production of 
goods and the provision of services under the sustainable environment. 

At the same time, the tactical administration is also responsible for providing logistics, manage resources 
and knowledge, and capture information, for and from, the strategic administration of the research, innovation 
and develop to implement strategies and operations for transforming the mission towards the vision, achieving 
objectives and goals under the framework of the organizational values and principles. 

In summary, tactical administration is responsible for planning, supplying, monitoring and functional and 
organizational controls of logistic, knowledge, communications, technology, economy, people and. 
performance around of the Administrative Management of the strategy, the tactic and the operation. The 
Strategic Administration is responsible to improve the system, produce plans, programs and projects of 
research, innovation, improvement and development as a whole, from the environment and functional studies, 
information on the organizational behavior specifications, intelligence processes, vision, leadership and 
direction. The Operative Administration produces and provides missionary goods and services. 


3. ORGANIZATIONAL MATURITY PROCESSES 

Under the guidelines of OLP as Enterprise Architecture with architectures of business, information, and 
technology, the Organizational Maturity Model of Processes - PEMM - (Figure 4) is a tool to observe, verify 
and validate communication specifications and functional performance of components, practices and 
resources, before, during and after, of the processes execution to prove hypothesis and determine knowledge 
for achieve strategic, tactical and operational goals. 

At this time, the Integrated Communication Diagnosis (COIN) is used to verify and validate inter -phases, 
channels, technology, and media, quality, quantity, connection, meaning, continent, content, transmission, and 
reception of messages (Figure 3) from emitters and receivers for doing preventive and corrective actions to 
improve organizational climate, environment, workplaces, and emotional communications between internal 
and external stakeholders of the organization. 
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Once the COIN is established, the Tactical Administration proceeds to make the organizational and 
functional assessment using the measurement methods of OPMM (Figure 4) for establishing the capacity and 
stability of enterprise architecture using benchmarking to verify conformities and unconformities supported by 
findings to develop projects within plans and programs according to the priorities of the organization. 
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Figure 1. Administrative Management View 


Figure 2. Administrative Management Components 
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Figure 3. Organizational communications Evaluation 


Figure 4. Organizational Processes Maturity 


4. AGILE SCHOOL MODEL 

The "Agile School" architecture has the following building blocks (Figure 5) [(Llamosa and Mendez, 2010), 
(Llamosa and Gomez, 2013), (Llamosa and Camacho, 2013), (Llamosa and Camacho, 2013)]: 

• The Administrative Management directs and manages the government and strategic thinking; the 
research, innovation and development of goods and services, technology, continent, content and educational 
assessments; the resources, logistical and tactical; and, the operation of education services. 
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• The Strategic Administrative manages, operates and monitors the research, innovation, 
improvement and technological development projects of goods and services for all areas of the educational 
organization. 

• The Tactical Administrative manages, provides and monitors all logistics resources, processes, 
technology, people, knowledge, intelligence, configuration, quality and infrastructure for all areas of the 
educational organization. 

• The Operational Administrative provides and monitors all educational services of the authoring, 
instruction and assessment. 
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Figure 5. Higher Education Processes Network 

The "Agile School" Architecture connects supplies, inputs, processes, resources (actors, technology, 
knowledge and infrastructure) and the products and beneficiaries. The processes are seen as a sequence of 
activities that start in the needs of external beneficiaries and end in results received by all beneficiaries. 

The Table 1 shows the building blocks, activities and practices. Each practice has a maturity level and is 
associated with the roles of Administrator (Leader and Manager), Teacher (Mentor, Coach, Instructor, Trainer, 
Facilitator and Assistant), Learner (Leader, Member, and Student), Appraiser (Author and Internal, External 
and Proper Evaluator), Beneficiaries (External Stakeholders), all configuration resources, and infrastructure. 
Each of the practices has products and these products can be goods or services and they should have results for 
each of the beneficiaries and stakeholders. 

The "Agile School" architecture competencies are composed by performance competencies of processes 
and performance competencies of resources. Each type of competency is broken down into units and elements 
of competency, performance criteria, knowledge, applications and evidence to certify skills and conformities. 
And each product is associated with tasks and results and there are a complete relation among processes, roles, 
resources, products and outcomes. 

With this inventory of the elements of competency and their products and terms of outcome or impact, the 
compliance, satisfaction and control measures are set, and consequently, it is computable the potential 
capability and actual capability to estimate and control the Architecture [(Jurney, 2014, Sheikh, and Alnoukari, 
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2012)] under criteria of truth to admit, build and study cognitive phenomena, for justifying how it is generated, 
built, defined and institutionalized knowledge. Furthermore, it is validated the knower, the object known and 
the conditions of the educative value chain [(Mansourov and Campara, 2011) and (Martin, 2001)]. 


Table 1. Process: Building Block, Activities, Practices (Maturity Level) of Agile School 


Processes 

Building Block 

Activities 

Practices (Maturity Level) 

Administrative Management 

Direction 

1 . Decision Analysis & Making (3) 

2. Causes Analysis (5) 

3. Educational Strategic (3) 

Management 

4. Requirements Management (2) 

5. Educational Planning (2) 

6. Educational Monitoring/Control (2) 

7. Risk Management (3) 

8. Education Quantification ( 4) 

Strategic Administrative 

Systemic Improvement 

9. Definition and Improvement (3) 

10. Research, Innovation and Development (5) 

Tactical Administrative 

Quality 

Assurance 

1 1 . Configuration of Knowledge Assets Management (2) 

12. Measurement and Analysis (2) 

13. Quality Assurance (2) 

14. Training (3) 

15. Communications Management (3) 

16. Performance Measurement ( 4) 

Operational Administrative 

Authoring 

17. Requirements Definition (3) 

18. Educational Design/Development (2) 

19. Educational Verification (3) 

20. Educational Validation (3) 

Instruction 

21. Instruction Service Delivery (2) 

22. Instruction Service Continuity (3) 

23. Incident Resolution/Prevention (3) 

Evaluation 

24. Evaluation (3) 


Formation & development Steps 
Forming 


’Intellectual Domain 
(Cognitive System) 


Directing and 
Training 


Coaching and 

Norming 


Mentoring 

•Practical Domain 

r 


’Emotional & 
Corporal Domain 
(Affective System) 


Storming 


(Affective System) 


Supporting and 
Facilitating 


Delegating 


’Creative Domain 
(Expressive System) 


Adjourning /Mourning 


Performing 


■Autonomous Domain 
(Integral System) 


Autonomy 

Situational Leadership 


Knowledge 

& 

Skills 


F 


RESEARCH BASED LEARNING 




PROJECT BASED LEARNING 4- 


-■> ANDRAGOGY < 


Delivery based Value-4- 


-► PROBLEM BASED LEARNING 


-►PEDAGOGY 


-►Stakeholders Engagement 



Practical Workteam Adapting Planning 


Problem Detection & Resoltion Continuos Improvement 


Figure 6. Teaching - Learning Process 
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Finally, it is important to note that each stage of the educational process requires to achieve a gradual 
iterative training in knowledge and skills and all phases of the teaching-learning activities covers (Button 
Figure 5 and Top Figure 6) (Barron et al, 2011): The Intellectual Domain (cognitive system) in which the 
learner receives direction and training; Emotional and Physical Domain (emotional system) in which the 
learner, with the support of an educational coach and a facilitator, publicly confronts his learned achieved in 
the cognitive system; The Practical Domain (emotional system) in which the learner, with the support of 
educational assistants and facilitators, get experience, skill and maturity as practitioners; The Domain 
Creative (expression system) in which the learner is delegate to perform own creations; and. The 
Autonomous Domain (Integrated system) in which the learner is free to make her own practices. 


5. AGILE SCHOOL PILOT 

With the "Agile School" architecture specified, a pilot was established in the Moodle platform of the 
Industrial University of Santander, and then, the provision of educational services of government, teaching- 
learning, consulting and advisory, was provided in the Information Centre of Technology and 
Communications and the CIDLIS. 

With the pilot produced from the "Agile School" architecture (Button Figure 6) they were established the 
following activities: 

1 ) It tested and validated the value chain of the educational process and its products with stakeholders, 
resources, activities, time required and products, and the outcomes were satisfactory (value commitment) and 
transcendental (management, problem solving, teamwork and improvement). 

2) The introduction of competencies of processes performance, resources and products were evidenced 
in processes assessment and validation, the optimal use of resources, the elimination of wastes and lead time, 
timely response to requests, continuity and control of processes, evolutionary and iterative delivery of product 
releases, validate methods and media of communication, achieving competencies, and, the measurement of the 
potential and actual capability level of processes, resources and products. 

3) It constituted the teaching-learning team: authors, instructors, facilitators, assistants, coaches, 
mentors, auxiliaries and students, and synchronously, the required products and activities, were proposed and 
were developed in each of the teaching-learning cases and projects in each of the educational process iteration. 

4) The educational process iterations involved the synchronization of processes, requirements and 
conditions of learning development by each case and project through instruction, executions, iterations, daily 
checks, verifications, validations, transferences, retrospectives, improvements and innovations. This exercise 
was fulfilled when it achieved the scope established for each activity. 

5) When the last cycle of execution of "case-project" was closed, it was necessary to do an overall 
retrospective for innovations, enhancements and improvements. 

The pilot (Figure 7) was applied to two courses to obtain an experimentation appropriate sample. The 
courses were: Quality Engineering (Top level: 7th semester) and Statistics and Probability for Engineers 
(Button level: 3th Semester) at "Electrical and Electronics Engineering School" of the Universidad Industrial 
de Santander. The students' activities were linked by work team of six students, two of the top-level course, 
and four of lower -level course. Both classes had a common interface module with statistics tools for 
synchronization and promotion of knowledge, skills, tools, techniques and abilities. Situation, which allowed a 
comprehensive teaching-learning project and cases in the domain of students and although being on different 
levels, they developed learning collaborative and work complementary with projects and cases covering a 
single purpose improving products and processes, achieving common scope. 

The directly and indirectly measurement system of processes, competencies and products included tools 
supported in rubrics and test to apply to reports, daily revision, exams and retrospectives, for each module, 
case and project of the courses whole content. The value chain included measurements of processing times, 
waiting times, and times of transition. On inputs of each phase were detected errors, defects, faults and 
failings, produced by the predecessor stages and it was detected products without providers and consumers. 

The products of the teachers, authors, mentors, coachers and students were checked by the beneficiaries, 
who in turn expressed their assessment, findings and suggestions in each case. 
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From the perspective of leadership, management and administration got in Agile School pilot (Figure 8): 

• Under the "planning - monitoring - control" under the "lean - agile" approach, the stakeholders were 
integrated with general and specific teaching-learning attainments of work teams. All achievements were 
evidenced through products and behaviors recorded as formal issues in each phase of the course. 

• All activities were motivated under the "serious game" approach for encouraging the listening, 
hearing, touch, and taste for the analysis, reasoning, and synthesis for achieving sustained behavior in 
emotional intelligence about "strategy - tactics - operation" supported by teamwork engaged for authoring and 
teaching - learning under the satisfaction perspective by the value creation and stakeholders commitment in 
results, and transcendence by problems management and continuous improvement 

Administrative Management 

Organizational Systems 




Figure 8. Courses Work Teams 


• The capacity of the courses (Figure 9) was above 71%. The result of the evaluation of the capabilities 
for each of the tested courses has generated improvements to our work environment. 



Figure 9. Performance Evaluation of students 
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6. CONCLUSIONS 

Lessons learned by the use of the Agile School Architecture enhances the Teaching-Learning process to 
create a dynamic, agile, and collaborative learning and integration with the environment, a fact that motivates 
and creates value for all those involved in the educational process. 

The achievements of the methodological performance of the "Agile School" architecture are translated into 
discipline to internalize the processes and principles of performance and accountability whereby by any action 
is made spontaneously by all teams, as an integral whole. 

Future work Agile School argues to extend the model to the scenarios of simulation to work components as 
educational units on which their structures and behavior is not known, in order to perform intelligence to 
create influenced collaborative environments using the paradigms of systems of systems. 
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ABSTRACT 

Emerging adult college students have developmental and educational needs which are unique to their phase of life. The 
purpose of this study was to examine academic and non-academic technology use by emerging adult college students. 
Survey results (N=235) provided insights into emerging adult college student technology preferences and frequency of 
use for academic and non-academic purposes. This study found that emerging adult college students have distinct 
technology preferences and practices relating to both academic and non-academic use. 


KEYWORDS 

Technology, Emerging Adult, College. 


1. INTRODUCTION 

Individuals between 18 and 25 years of age are increasingly considered to be in a unique developmental 
period called emerging adulthood ( Arnett, 2000). This phase of life between adolescence and adulthood is 
“not merely a transition but a separate period of the life course,” with distinct features and needs (Arnett, 
2007, p. 69). “In this developmental phase students need environments that offer relevance, revelation, 
responsibility, and relationships” (Flowers, 2014, p. 1). Flowers (2014) explains experiences are needed that 
reach these individuals where they are, yet stretch them personally and academically, making them take 
responsibility for their future. Key in the learning process are the relationships between educators and 
students; these social connections are impacted by technology. Many of emerging adult relational 
connections are conducted via technology. Levine and Dean (2013) point out fundamental changes in today’s 
emerging adult college students based on their connections to technology. Previous research efforts assess 
attitudes and uses for technology in instruction generally but do not delineate differences between academic 
and non-academic technology preferences in emerging adults. Understanding the role technology plays with 
this emerging adult age group, who are often dubbed “Digital Natives” or the “Net Generation” (Bennett & 
Maton, 2010; Prensky, 2010; Tapscott, 1998), is also key in understanding the learning process in the 
emerging adult developmental period. This paper contributes to a deeper understanding of collegiate 
emerging adult development and learner needs. 


2. LITERATURE REVIEW 

2.1 The Use of Technology by Emerging Adult College Students 

The active and significant role technology plays in most emerging adult lives is easily observed and has been 
documented (Prensky, 2010). Technology permeates most aspects of the emerging adult’s life; however, 
certain technology use is sometimes considered a distraction, resulting in debates concerning the appropriate 
place for technology use (Jackson, 2013). Much current research involving emerging adult college students’ 
technology use is centralized on isolated populations and specific arenas of the college context. For example, 
Cassidy, et al. (2011) focused their study of technology use and preference around library services. Other 
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researchers narrowed their studies to specific technological resources, formats, or applications such as 
mobile/cellular devices (Baker, et al., 2012; Geng, 2013); social networking sites (Subrahmanyam, Reich, 
Waechter, & Espinoza, 2008); and online video use (Sherer & Shea, 2011). 

Institutions of higher education are increasingly integrating technology into all aspects of academia. 
Despite increased use of technology in today’s collegiate setting, the degree and effectiveness of technology 
use varies. Most colleges now use some version of a virtual learning environment (VLE) or course 
management system such as Blackboard (Morgan, 2003; Weller, 2007). Pitler, et ah, (2007), indicate that the 
effective use of technology often serves to increase student learning, understanding, motivation and 
achievement. Baker, et ah, (2012), however, investigated whether mobile devices belonged in the classroom 
despite their commonplace use in emerging adults’ everyday life. Educators are increasing turning their 
attention to technologies and how they can or should be incorporating it into the collegiate learning arena. 
Determining effective use of technology is actually contextually dependent (Owston, 2006). Understanding 
student perspectives on technology use will aid in knowing the learner and how to effectively educate them. 
The purpose of this study was to examine academic and non-academic technology use by the emerging adult 
college students, which provides perspective upon the emerging adult learners and the role of technology in 
their educational pursuits. 

2.2 Developmentally Appropriate Instruction for Emerging Adults 

Most emerging adults experience some education or training beyond high school, as noted by the Clark Poll 
where 79% of participants had some college or vocational experience (Arnett & Schwab, 2012). Most 
institutions of higher education have adapted systems of communication, processing information, retaining 
records, etc. based upon changes in technology. This transformation to increased technology use is a surge 
not limited to administrative decisions. The impact of technology has also changed how these institutions 
provide instruction. Technological resources and tools have transformed how and where content is delivered; 
however, understanding the needs of the learner is still a key issue. The means for instruction at any level 
needs to be developmentally appropriate (Gonzalez -DeHass & Willems, 2013, Gorra, et al., 2010). 

2.3 Technology and Collegiate Emerging Adults 

There is no doubt that collegiate emerging adults today have had a great deal of exposure to technological 
advances both personally and educationally. Internet and mobile technologies are being used for accessing 
information and communicating (Dahlstrom, Walker, & Dziuban, 2013; Maton & Bennett, 2010). In many 
cases, technological devices are used routinely in their personal lives to the extent that these individuals 
cannot imagine life without the device. As new technologies have emerged, they have changed patterns of 
student interactions, entertainment, time use, and even the use of their campus spaces and facilities (Levine & 
Dean, 2013). The higher education instruction can be delivered from virtually anywhere and at any time and 
either in online or blended formats (Gorra et al., 2010). Some variation in technological access exists as a 
result of social-economic barriers; however, that gap continues to diminish as the use of mobile devices 
increases. Technology use among collegiate emerging adults is now expected in college settings (e.g. online 
admission applications, communicate via college provided email, online course registration). Additionally, 
most colleges and instructors utilize course management systems such as Blackboard. Technology use has 
transformed education, however, actual technological preferences and patterns of emerging adult students for 
academic purposes needs to be examined. 

2.4 Theoretical Framework 

The conceptual and theoretical framework for this study rests upon the constructivist concept of learning by 
building knowledge, but more specifically, uses Vygotsky’s theme which considers sociocultural interaction 
as key in the learning process (Gonzalez-Dehass & Willems, 2013; Vygotsky, 1978). According to Vygotsky, 
tools are objects from a person’s culture which increase learning through problem solving (Gonzalez-Dehass 
& Willems, 2013). Technology can be viewed as such a tool for emerging adults in collegiate settings. 
Sociocultural interaction using technological tools is further grounded by the research-based APA Learner- 
Centered Psychological Principles, which emphasize the context of learning as an essential factor influencing 
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learners. “Learning is influenced by environmental factors including culture, technology, and instructional 
practices” (APA Work Group of the BEA). 

Bennett and Maton (2010) suggest emerging adult college students make choices about their technology 
use for highly contextualized purposes, influenced by life stage and interest. Individuals live within many 
contexts (i.e. social and familial, educational, vocational) which serve to create a contextual dependency 
upon specific related technology (Levine & Dean, 2013). The educational context serves to support 
technology use with certain technological tools and resources specifically for academic purposes. At the 
same time, technology is undoubtedly being utilized for non-academic purposes. When technology use by 
emerging adults is viewed through a contextual framework, nuances for practice and preference may emerge, 
which will impact learning. The following research questions address emerging adult use of technology in 
the context of the college setting: 

1) What technological tools and resources do emerging adult college students prefer for academic 
versus non-academic purposes? 

2) Do demographic variables impact technology preferences? 

3) Does the amount of time in a week devoted to academics relate to technology use? 


3. METHODS 

This study was descriptive comparative in nature and used survey methodology following suggestions of 
Busha and Harter (1980) in which a representative sample was solicited, questioning was cautiously designed 
for clarity and avoidance of bias or negativity, and was relevant to the emerging adult developmental state 
and situation. This survey methodology is useful in describing the emerging adult practices of academic and 
non-academic uses of technology. 

3.1 Survey 

Data was collected through an IRB approved, self-report internet survey administered via Survey Monkey. 
Participants anonymously completed the survey within an 8 to 10 minute timeframe and then were invited to 
email the researcher if interested in being entered into a random drawing for $25 gift cards. The survey 
consisted of 19 questions utilizing check-off boxes and ranking for Likert-type scaled responses indicating 
preferences and frequency. Open-ended response boxes were also provided for additional comment. 

Survey questions were designed with input and consensus from a panel of emerging adult students. 
Further content validity was established using a focused literature review. Some survey questions sought 
demographic information concerning students’ genders, majors and ethnicity. This questioning followed 
practices by researchers such as D’Angelo and Woosley (2007) who found differences in technology 
preferences based upon the student’s major, and Baker, et al (2012) who noted that gender affected 
technology perceptions of in-class technology use. Such questioning investigated whether demographic 
variables impacted collegiate emerging adult technology use for academic versus non-academic purposes. 

3.2 Participants 

Participants were students at a private liberal arts college in the northeastern United States who were invited 
to complete a survey about their academic patterns and technology use. Those who chose to complete the 
survey (N=235) represented approximately a 10% response rate and were spread proportionately across class 
years. Respondents were 72.8% female and 25.5% male, while 1.8% indicated other for gender or chose not 
to answer (See Table 1). Respondent ethnicities included Caucasian (77.9%), Asian (10.6%), Hispanic 
(7.7%), African American (4.7%), and other (4.3%). 

Survey respondents (N=235) were mostly self-identified as female (N=171, 72.8 %), which is higher than 
the national and local trend for this demographic. According to the National Center of Education Statistics 
(US Department of Education, 2012), 57% of college students in the United States in 2010 were female and 
this particular college reported a 59% female population in 2013. The gender representation of participants, 
however, was distributed throughout class years with a standard deviation of participants across class year for 
males 4.47 and females 7.54. 
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Table 1 . Demographics. The numbers in this table represent per cent from the total survey response (N=235). Gender: 
M-Male, F-Female, O-Other; Cultural Ethnicity: AA-African American, AS-Asian, CA- Caucasian, HI-Hispanic, NA- 
Native American, OT-Other. PNA: prefer not to answer. Note: Some respondents choose more than one cultural 

ethnicity. 


(Class year, gender, and ethnicity demographics) 


Class 

M 

Gender 
F 0 

PNA 

AA 

Cultural 
AS CA 

Ethnicity 
HI NA 

OT 

PNA 

Freshman 

13 

43 

1 

1 

3 

7 

48 

6 

2 

2 

1 

Sophomore 

19 

37 

0 

1 

4 

9 

41 

4 

0 

1 

0 

Junior 

17 

29 

0 

1 

0 

3 

37 

5 

0 

3 

2 

Senior 

11 

62 

1 

0 

4 

6 

57 

3 

0 

4 

2 

Total 

60 

171 

2 

3 

11 

25 

183 

18 

2 

10 

5 


3.3 Data Collection Procedure 

Students attending a liberal arts college in northeastern United States were sent an invitation through the 
college email system encouraging participation in an online, anonymous survey. The incentive for 
participation was an optional entrance into a random drawing for $25 gift certificates. One week later, a 
reminder email was sent inviting any to participate who had not already done so. The first invitation gleaned 
177 responses over a four day period and the second resulted in an additional 58 responses over a two day 
period. Data results were then examined for descriptive and comparative analysis. 


4. RESULTS 

The results of the study are reported in response to the following research questions: What technological 
tools and resources do emerging adult college students prefer for academic versus non-academic purposes ? 
Do demographic variables impact technology preferences?, and Does the amount of time in a week devoted 
to academics relate to technology use? 

4.1 Academic versus Non- Academic Use of Technology 

This study investigated technological tools and resources emerging adult college students prefer for academic 
versus non-academic purposes. The researcher first analyzed the tools which served as access points for 
technology use by emerging adult college students and how often those specific tools were used (Figure 1). 
Survey results indicated respondents’ personal computers were used most daily for academic purposes (95%) 
but cell phones were used the most daily for non-academic purposes (97%). Results also indicate that iPads 
and tablets are being used less frequently as computers for academic purposes; 77% report never using an 
IPad/tablet for academics and 73% never use them for non-academic as well. It could be theorized that iPads 
and tablets are less accessible resources while mobile phone devices are becoming increasingly accessible 
and versatile. Additionally, students report daily use of institution-owned computers which helps address 
access issues that may arise due to socio-economic status. 
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Technology Devices Used Daily 
Academic purposes Non-academic purposes 


Institution computer - 19% 


Personal computer - 95% 


Cell phone - 56% 


iPod/MP3 Player - 16% 


iPad/Tablet - 6% 


Institution computer - 12% 


Personal computer - 96% 


Cell phone - 97% 


iPod/MP3 Player -50% 


iPad/Tablet - 9% 



Figure 1 . Academic and non-academic daily use of technology devices 

The results of the survey indicate that technological applications and resources used are contextually 
dependent. Participants in this study rarely used online resources for academic purposes. As displayed in 
Table 2, distinct differences are evident when comparing technology applications and resources based upon 
use for academic or non-academic purposes. 

Table 2. Academic and non-academic frequency of use for technological applications and resources. 
Frequencies of use for academic and non-academic technology purposes 




Academic 



Non-academic 


Online Resources 

Never 

Monthly 

Weekly 

Daily 

Never 

Monthly 

Weekly 

Daily 

Online T.V. 

138 

68 

19 

6 

11 

36 

87 

100 

Online Music 

151 

31 

23 

27 

28 

25 

59 

122 

Online Storage/ Sharing 

76 

78 

45 

34 

98 

68 

37 

32 

Online Presentation Tools 

40 

99 

60 

33 

144 

48 

33 

9 

Online Photo Sharing Sites 

201 

23 

6 

3 

163 

44 

18 

10 

Online News 

59 

75 

65 

33 

50 

52 

62 

71 

Online Applications 

Email 

0 

0 

18 

216 

4 

14 

49 

167 

Skype / Face Time 

179 

36 

17 

0 

44 

97 

68 

25 

Facebook / Google + 

87 

46 

60 

40 

19 

3 

11 

199 

Linked-In 

161 

47 

17 

8 

137 

49 

34 

13 

Twitter 

206 

12 

11 

4 

154 

16 

26 

39 

SnapChat / Instagram 

201 

7 

7 

17 

67 

14 

35 

118 

Tumblr / Pinterest / Reddit 

196 

18 

13 

5 

108 

37 

31 

58 

Blogs / Wiki spaces 

170 

34 

22 

7 

145 

37 

34 

19 

YouTube 

76 

89 

50 

18 

8 

22 

107 

98 

TED talks 

99 

103 

21 

9 

72 

108 

41 

12 

Google Maps/Earth, GIS, etc. 

138 

70 

14 

9 

78 

90 

54 

12 

Simulations/Games 

195 

24 

8 

4 

115 

49 

40 

30 


Of the emerging adult college students who responded to this survey (N=235), many noted that they 
never use certain resources for academic purposes, such as online T.V. (N=138, 59%), online music (N=151, 
64%), and photo sharing (N=201, 86%); yet, for non-academic purposes these same items are often used 
daily. This same population reports frequent non-academic use of online TV (43% daily) and online music 
(52% daily). The most heavily used weekly online academic resources were online news, presentation tools, 
and storage/sharing, conversely, these were rarely used for non-academic purposes, with the exception of 
news. 

Social media applications such as Facebook, Google+, SnapChat, and Instagram were heavily used for 
only non-academic purposes. This indicates a preference for keeping social and academic technology use 
separate. Even YouTube which carries the possibility for instructive purposes was favored as non-academic. 
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Email is the only application/resource that seems to transcend contextual use boundaries with 92% of 
students using it daily for academic purposes and 71% for non-academic purposes. It could be argued that 
contextual boundaries are less evident in this resource because of the prevalence of colleges requiring 
communication via email, and additionally the rise of mobile devices upon which email can be received. 

The results suggest many technological items used are compartmentalized for specific contexts and 
students struggle to see the usefulness and applications of these resources outside of the assumed parameters 
in which they are used. Possibly, educators have not broadened students’ perspectives by modeling 
technological usefulness for many resources and application. 


4.2 Demographic Variables and Technology Use 

The demographics examined for this study were class year, gender, cultural ethnicity and major as noted 
previously in Table 1. Class year and gender had normal distributions as compared to most liberal arts 
college campus in northeastern United States. Cultural ethnicity consisted of 78% Caucasian, thus, there was 
not enough variance to provide significance between participant responses. 

Gender impacted the amount of time spent outside of class on academic purposes. Female students spent 
14% more hours outside of class weekly on academics than males, with an average of 43.5% of that time 
using technology. Both males (76%) and females (89%) preferred their actual course texts and non-academic 
reading to be in print rather than digital format. 

Males and females equally ranked in-person communication as their first preference (50% each) for 
academic communication; however, more females (85%) than males (65%) prefer non-academic 
communication to be face-to-face. Both males and females ranked email as their next preferred manner to 
communicate about academics, but for non-academic communication only 2% of males indicated preferring 
email. Females indicated texting (63%) as their preferred type of non-academic communication. 


4.3 Time Spent on Academics as Related to Technology Use 


Emerging adults have many activities competing for their time each week. The survey revealed that 59% of 
students spend between 6 to 15 hours weekly on academics outside of class time (Figure 2). 


21+ HR 1-5 HR 
13% 7 °/° 


16-20 HR 
21% 


6-10 HR 
28% 


11-15 HR 
31% 


Figure 2. Hours per week spent on academics outside the classroom. 

Those same individuals note 50% or more of their academic time involves using technology (Table 3). 
Another 34% of these emerging adult college students spent 16 or more hours a week devoted to academics 
outside of classes, and also use technology approximately 50% during those hours. Increases in time devoted 
to academic purposes are directly associated with technology use. 
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Table 3. Percentage of Academic Time Involving Technology. Emerging adult students were grouped by the amount of 
time they recorded spending on academics outside of class. Hours were tallied for how much of that time spent on 

academics involved the use of technology. 


(Academic time outside of class spent using technology) 


Per Cent of Academic 
Time Involving 
Technology 

Group 1 
1-15 acad. hrs. 
(N=73) % 

Group 2 
16-20 acad. hrs. 
(N=49) % 

Group 3 
21+ acad. hrs. 
(N=30) % 

0-25 

0 

0 

0 

0 

0 

0 

26-50 

16 

22 

8 

16 

4 

13 

51-75 

33 

46 

24 

49 

15 

50 

76-100 

23 

32 

17 

35 

11 

37 


Many participants spent less than 15 hours a week on academics outside of class (N=73, 31%), however, 
78% of that time involved using technology. Those who spent more than 20 hours a week on academics 
reported using technology 87% of that time. Increased academic time revealed increased technology use. 


5. CONCLUSION 

One limitation in considering the study results is that these emerging adult college students are centralized on 
one campus, in one region of the United States, and thus the results are generalized to this specific population 
or similar sub -populations. Further research is needed with a more diverse group of emerging adults both 
ethnically as well as from differing regions of the United States and internationally. Such comparison would 
provide more understanding of developmental appropriateness and technology use with emerging adults. 

Frequency patterns for technology use indicate emerging adult students have daily technological 
connections and that technology plays a role in both academic and non-academic contexts. Some slight 
differences exist between genders and are less noticeable in regard to class year or major demographics. 

This study revealed an important implication for educators of emerging adult college students, survey 
participants use institution-owned technology devices, specifically college -owned computers or laptops, on a 
weekly or at least minimally, monthly basis. This serves a social justice purpose. Continuing to offer 
opportunities for all students to have access to technology enables those who may not otherwise have the 
resources to utilize technology for learning purposes. 

In conclusion, this study revealed the phenomenon that emerging adult technology use is contextualized. 
These individuals struggle developmentally to transfer their technological abilities from one context to 
another, and thus, may struggle to conceptualize the academic potential of certain technological resources 
and applications for academic purposes. The researcher challenges educators of emerging adult college 
students to discuss technology application with students, and incorporate varied technology into collegiate 
pedagogical practices. 
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ABSTRACT 

The process of identifying techniques for fostering creativity, and applying these theoretical constructs in real-world 
educational activities, is, by nature, multifaceted and not straightforward, pertaining to several fields such as cognitive 
theory and psychology. Furthermore, the quantification of the impact of different activities on creativity is a challenging 
and not yet thoroughly investigated task. In this paper, we present Semantic Lateral Thinking as a candidate for fostering 
creative thinking within storytelling educational activities. We describe a set of tools implementing Semantic Lateral 
Thinking techniques over storytelling, and we present Creative Stories, a digital storytelling game incorporating these 
tools. Furthermore, we propose a metric for quantifying creativity within the game, leading to the evaluation of the 
creativity exhibited in the stories produced within the game in relation to the usage of the Semantic Lateral Thinking 
stimuli provided by the relevant tools. 


KEYWORDS 

Digital Educational Games, Creativity Metrics, Semantic Lateral Thinking. 


1. INTRODUCTION 

“Human-creativity is something of a mystery, not to say a paradox”, states Boden in her book The Creative 
Mind (Boden, 2004), when introducing us as to the ‘what’ and ‘how’ of creativity. Apart from unveiling the 
mystery of human creativity, i.e. the ability to come up with ideas or artefacts that are new, surprising, and 
valuable, she also discusses how computers can help us understand it. 

Along with such philosophical approaches, research results from neuroscience should also be considered 
in the process of revealing/ understanding the Human Creative Process. Such an example is the work of 
(Limb and Braun, 2008), who examine how the human mind perceives complex auditory stimuli e.g. music. 
In this case, they look at the brains of improvising musicians and study what parts of the brain are involved in 
the kind of deep creativity that happens when a musician is really in the groove. Their research has deep 
implications for the understanding of creativity of all kinds. (Nachmanovitch, 1990), an improvisational 
violinist, computer artist and educator, in his book Free play states that creativity arises from bricolage, from 
working with whatever odd assortment of funny-shaped materials we have at hand, including our odd 
assortment of funny-shaped selves. 

In the process of involving machines in the creative work, (Lubart, 2005) includes the case of Human- 
Computer cooperation during idea production and proposes a creative thinking strategy, which relies on 
random or semi -random search mechanisms to generate novel, unconventional ideas. The role of machines in 
this case is to implement random searches that challenge humans in the process of selecting/ generating new/ 
innovative ideas and perhaps turning them into creative products. In this context, the Semantic Lateral 
Thinking theory is particularly well-suited to establish the cooperative framework, by implementing 
automated components that adhere to the theory and applying them to a suitable educational medium, such as 
open-ended digital games. In this paper, we discuss on the core characteristics of the Semantic Lateral 
Thinking theory, describe its application in a digital storytelling educational game and present metrics that 
help us quantify the impact of the overall process on fostering the creativity of the participating players. 
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The paper is structured as follows: Section 2 discusses on some of the techniques proposed by the 
Semantic Lateral Thinking theoretical framework for fostering creativity. Section 3 briefly presents the 
application of those Semantic Lateral Thinking techniques in storytelling activities via the usage of 
appropriate computational tools. Section 4 showcases Creative Stories, a storytelling game that incorporates 
tools for introducing these tools in a gamified environment. Section 5 presents the scoring mechanism 
incorporated in the Creative Stories game, intended on quantifying the perceived creativity within the game. 
We conclude and indicate our future steps in Section 6. 


2. SEMANTIC LATERAL THINKING (SLT) TECHNIQUES 

The term Lateral Thinking was invented by (De Bono, 1970). It adheres to the tendency of self-organizing 
systems, such as the human brain, to form and move across asymmetric patterns. Tools and processes 
supporting lateral thinking aim to assist that “lateral” movement, providing the means to escape from a local 
optimum in a thinking process towards a more global optimum. 

Semantic Lateral Thinking (SLT) involves the use of different conceptual Po (De Bono, 1972), (De Bono, 
1990), a tool or an operator meant to provoke and dislocate from habitual patterns and forms, as well as 
disassociate established connections. Several techniques can support SLT e.g. Random Stimulus, and/or Re- 
conceptualization. 

The main principle of the Random Stimulus technique is the introduction of a foreign conceptual element 
with the purpose of disrupting preconceived notions and habitual patterns of thought. The human actor is thus 
enforced to integrate/ exploit the foreign element in the production of a solution/ idea, and bring together 
disparate domains. 

Randomness is the main guarantor of foreignness and hence of stimulation of creativity. Foreignness in 
this context has two main dimensions: (a) It is important that the human actor feels that he/ she has to 
somehow integrate/ exploit an element which is introduced completely from without, whose introduction is 
in no way under the his/ her control. In some ways an intruder has to be re -conceptualized as a friendly aid; 
and (b) the new element should, at least initially, be as unconnected as possible to the subject/ type/ structure 
of the problem. By doing so, we someway ensure that no unconscious/ unobserved pre-established analogies, 
preferences and connections creep in the selection of the stimulus. After the presentation of the problem, one 
is asked to use creatively in the reasoning process the random stimulus provided. 

Re-conceptualization involves the use of already established solutions and ideas in new environments. 
One is encouraged in exploiting the potential of familiarity in the production of novel ideas. The familiar 
features of the established solution/ idea will re-inscribe themselves on the unfamiliar environment or appear 
in a new light. 

The core distinctive characteristics of the SLT theory -randomness, introduction of external stimuli and 
re -consideration of an idea in a new environment- constitute digital educational games as a highly relevant 
platform for implementing and testing the effectiveness of the theory on fostering human creativity. The rest 
of the paper presents the application of the aforementioned SLT techniques in a storytelling game, via the 
usage of relevant computational tools, and showcases the proposed foundation for measuring its effectiveness 
on the attempt to foster creativity. 


3. INCORPORATING SEMANTIC LATERAL THINKING IN 
STORYTELLING ACTIVITIES 

In this section, we briefly present a set of computational tools that transparently support Semantic Lateral 
Thinking techniques. These tools are focused on textual information, that is, the provided elements are words 
or phrases that act as the random / external stimulus for the humans involved in the activity. The underlying 
semantics and contexts of these words, are to be analyzed and lead to alternate paths of thought, thus 
fostering out-of-the-box, creative thinking. 
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3.1 Thinking Seeds Generator 

The Thinking Seeds Generator provides a textual stimulus, having a varying semantic distance from its input. 
The produced word, as it is semantically distant from the initial state, is meant to act as an initiative to think 
out-of-the-box, re-contextualize ideas or be led to examine other perspectives of a problem / situation. 

The input of the Thinking Seeds Generator is a seed phrase and a difficulty degree, which denotes the 
semantic distance between the random words that will be returned and the initial phrase. In this context, the 
semantic distance of two terms is the number of edges in the WordNet [Fellbaum, 1998] synset graph that 
must be traversed in order to reach to a synset starting from another- specific- synset. 

The initial word to be used for the process is determined depending on the size of the textual input. When 
the input is a single word or a phrase up to three words, the input is processed as it is. In the case of larger 
texts, the service discovers the dominant terms within the text as follows: 

1 . Stopwords are removed from the text; 

2. The remaining words are stemmed, and hashed with respect to their stem; 

3. The three (3) most frequent stems are considered as dominant and the words having these stems are 
considered the dominant words within the text; 

4. One of the dominant words is selected randomly as the seed to be used. 

Following the process of determining the seed word, the service traverses the WordNet graph according 
to the following methodology: 

1. Retrieve the WordNet synsets to which the seed word belongs; 

2. Start from the synset containing the most words; 

3. Select the word in the selected synset that belongs to the most synsets; 

4. Repeat the first two steps for the selected word until the number of steps is equal to the set 
difficulty; 

5. Select all the words belonging to the last visited synset. 

6. Randomly pick one of the words belonging to this synset. 

3.2 Web Miner 


The Web Miner is used to provide a summary of web content that is related to an input text segment of 
variable size. The summary is expressed as a tag cloud structure, i.e. the service returns a set of dominant 
words found in the examined web content, along with their frequency of appearance. 

The input of the service is a word or short phrase, and an indicator that specifies if the service should only 
handle content safe for children. The service invokes a Search Engine wrapper and retrieves the HTML 
content of the first 50 results returned by the search engine. The content is cleaned using the boilerpipe 
library [Boilerpipe, 2014] in order to obtain the textual content of these pages. The stopwords present are 
removed and the remaining content is stemmed -using the Snowball stemmer [Snowball, 2014] - and hashed 
in order to calculate the TF-IDF value for each distinct stem. 


Let P the number of pages returned from the Search Engine wrapper, p an individual page in P and t a 
distinct stem. The weighted frequency of t in P is: 

Snep t/(t, p) X idf(t, P ) 

WF(t,P ) = p \ P \ L ^ J — L 

where tf is the logarithmically scaled term frequency of term t in a page p 

t/(t,p) = log (f(t,p) + 1) 

and idf is the inverse document frequency of this term in set P 

\P\ 

idf(t,P ) = log — - 

S |{p£P:t£p}| 

For each distinct stem, the most frequent form (with respect to its raw number of occurrences) is chosen 
to build a structure that encapsulates the [stem form, weighted frequency] pairs for the entire content. 
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3.3 Cloud of Thoughts 

The Cloud of Thoughts service provides a summary of a text segment, by examining the dominant words / 
short phrases found within the segment and returning them as a tag cloud structure. Its aim is to identify and 
present the major ideas present in the text, giving to the user a synopsis of others’ thoughts that can lead him / 
her to change thinking perspectives, guiding his thought in a different path. 

The service is invoked with the text to be summarized as its input. After the removal of stopwords, it 
calculates the logarithmically scaled term frequency as shown above. Finally, a structure that encapsulates 
the {dominant stem form, term frequency} pairs is returned. 

3.4 Competitive Thinking Spaces 

The Competitive Thinking Spaces service relies on the premise that a text segment may contain different 
aspects / points of view and the user can focus on a specific one to proceed with a line of thought. Thus, the 
service analyses a text fragment and identifies different groupings of the concepts included in the fragment, 
returning them to the caller. 

In order to determine the thinking spaces, the service operates on a text segment provided as input. It 
discards stopwords and then clusters the obtained word set. If the produced clusters exceed a specified 
number (e.g. 4), the service reduces the clusters to this number, using the distances between the clusters. It 
finally returns to the calling agent a structure that encapsulates the clusters and the words / phrases belonging 
to each cluster. 


4. THE CREATIVE STORIES GAME 

This section provides an example on the usage of the described computational tools in the context of Creative 
Stories, a storytelling game that uses the various tools in a gamified environment. We firstly present the setup 
phase for a Creative Stories game session. We proceed to demonstrate the execution of a Creative Stories 
game session and present the usage of the computational tools within the game. 

4.1 Creative Stories Session Setup 

The teacher defines the groups that will participate in the Creative Stories game session. He/ She defines the 
number of groups that will participate in the game session. 

The next step is to define the parameters of the actual game that will be used for the game session. The 
teacher defines the story’s theme, the range of difficulty for the input from the computational tools, and the 
way that the difficulty will progress during the game. Finally, the teacher can select the type of input from 
computational tools that will be used within the game. The Creative Input option will activate the Thinking 
Seed Generator and the Web Miner, while the Competitive Thinking Spaces option will activate the 
eponymous computational tool. 

After the teacher has setup the described parameters, the game session can be activated and the students 
can enroll as members of their group and play the game. 

4.2 Creative Stories Game Play 

After enrolling in the game session, the players are presented with a multi -panel environment from which 
they can provide their input, observe the activity of the other groups and get feedback from the computational 
tools and the teacher. The central panel presents the story fragments created so far by the group (Group 2 in 
the example) and contains the input field for writing and submitting a new story fragment, along with an 
indication for the points that will be added for the specific fragment, as they are calculated by the relevant 
computational tool (analyzed in section 5 of the paper). In the right-side panel, the players can see the 
progress of the other teams participating in the game session, along with tag clouds that summarize the 


138 



1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


stories of the other groups and can be used as inspiration and guidance for progressing with the story. These 
tag clouds are created via the usage of the Cloud of Thoughts tool, called with each group’s story as input. 

In the left-side panel, the players can observe their current score and use the input from the computational 
tools for obtaining input to be used within their story. There are two distinct modes of playing the game with 
respect to the type of automated input that is used, the Creative Input and Competitive Thinking Spaces 
modes. The next subsections briefly describe the characteristics of each mode. 

4.2.1 Creative Input 

Figure 1 depicts a mock-up of the game screen in the case that the Creative Input option was selected by the 
teacher. 


Creative Stories - Student View - Play (Creative Input) 
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C reative Points 


•Creative Input 

Thinking Seed 

^ museum ^ 


- 2 Points 


•Web Cloud - 



■ 8 Points 


^Change Difficulty ^ 


My Story (GROUP 2)- 


Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do 
eiusmod tempor incididunt ut labore et dolore magna aliqua. 


Ut enim ad minim veniam, quis nostrud exercitation ullamco 
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Figure 1. Playing Creative Stories in the Creative Input mode 


In this mode, the tools used for providing input to the players are (a) the Thinking Seeds Generator and 
(b) the Web Miner. In (a) the players are called to use the word or phrase provided by the Thinking Seeds 
Generator in their story fragment. In (b), the players are called to use all the words included in the tag cloud 
produced by the Web Miner in the story fragment. Each group is free to modify the difficulty (semantic 
distance) of the provided input and retrieve a different set of thinking seeds and tag clouds by hitting the 
refresh button. 


4.2.2 Competitive Thinking Spaces 

Figure 2 showcases the game screen in the case that the Competitive Thinking Spaces mode is active. In this 
case, the game uses the Competitive Thinking Spaces tool to provide additional input to the players. The 
input for the tool is the accumulation of the story fragments produced so far by all the participating groups. 
The groups try to use every word within one thinking space in order to “conquer” the respective space. When 
this is accomplished, the particular space becomes unavailable for the remaining groups, which have to focus 
on a different space. 
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Play 


Wrap-up 


Figure 2. Playing Creative Stories with Competitive Thinking Spaces 


5. CREATIVITY METRICS WITHIN CREATIVE STORIES 

Creative Stories incorporates a formulization of traditional Computational Creativity metrics, namely 
Novelty (Lehman and Stanley, 2008), Surprise (Maher, 2013), and Impressiveness (Lehman and Stanley, 
2012), in order to apply them over the textual artefacts obtained via the game. More specifically. Novelty is 
defined as the average semantic distance between the dominant terms included in the textual representation 
of the story, compared to the average semantic distance of the dominant terms in all stories. Surprise has a 
strong temporal dimension, which leads to the examination of the story’s method of continuation between its 
distinct fragments. To this end, we conceptualize surprise as the average semantic distances between the 
consecutive fragments of each story. Impressiveness has two main constituents. Rarity and Recreational 
effort. To model rarity, we compute the clusters of terms in each story. Following the same approach applied 
to the computation of the novelty metric, we calculate the semantic distance between the individual clusters. 
In order to provide an estimation of the rarity for the examined story, we calculate the sum of weights on the 
min-weight closure of the cluster graph compared to the maximum sum of weights in the story set of the 
group. On the other hand. Recreational Effort is calculated as the number of different clusters that each story 
contains, compared to the maximum number of clusters found in a story of the group. 
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During a Creative Stories session, the participating groups are rewarded with Creative Points, determined 
by their usage of input from the computational tools, as well as, usage of information from the activities of 
the other players. The Creative Points are defined as the product of the base Creativity Points returned by the 
Creativity Points Computation service and a modifier that depends on the usage of the aforementioned 
elements. 

We use two distinct functions for calculating the Creative Points in Creative Stories, depending on the 
type of input selected for the specific Creative Stories session. 

In the case that the Creative Input option is selected, the Creative Points are given by the following 
equation: 

/ n t n w \ 

/ 1 — T t difficulty A t — r / n * difficulty ; \ \ 

CreativePoints(T ) = I 1 + j j (^1 H — J + n + j j (^1 H — J — 2 N 0 1 BasePoints(T ) 

In the equation, N T is the number of times the player used the Thinking Seeds Generator services, N w is 
the number of times the player used the Web Miner services, while No denotes the number of words that the 
player used and appear on the tag cloud created from the other players’ stories, n is the number of words 
included in the tag cloud returned by the Web Miner. 

In case the Competitive Thinking Spaces is used as the computational tool input for the game session the 
equation for the calculation of the assigned Creative Points is the following: 

1 

CreativePoints(T ) = N C i usters BasePoints(T ) — -1V 0 

where, N ousters is the number of clusters completed by the specific team, and N„ is the number of words 
that the team used from the tag clouds summarizing the stories of the other teams. 

It is expected that the inspection of the values on the traditional creativity metrics and their comparison 
with the creative points score associated with each story will reveal correlations between the perceived 
creativity exhibited within the stories and the usage of SLT stimuli by the players. In this way, we can reveal 
the impact of the SLT techniques on the creativity of the Creative Stories players and the degree to which the 
introduced stimuli urged them to be more creative. 


6. CONCLUSION 

In this paper, we presented Semantic Lateral Thinking (SLT) techniques suitable for fostering creativity, 
which can be used in storytelling educational activities. We defined a set of computational tools facilitating 
the implementation of the aforementioned techniques in a digital storytelling game. Finally, we demonstrated 
the mechanics of such a game. Creative Stories, which builds upon the usage of a Creative Scoring 
mechanism for quantifying the impact of the SLT stimuli within a story. 

Our future work will focus on examining the correlation between the usage of these stimuli and the 
creativity as perceived by using traditional creativity metrics. Towards this objective. Creative Stories will be 
used in real-world educational settings, and the obtained results will be analyzed in order to assess the 
effectiveness of SLT on fostering creativity via its usage within digital educational games. 


REFERENCES 

Boden, M., 2004. The Creative Mind, Routledge, London, 2 nd Edition. 

Boilerpipe-Boilerplate Removal and Fulltext Extraction from HTML pages, https://code.google.eom/p/boilerpipe/, [June 
2014] 

De Bono, E. (1970). Lettered thinking: creativity step by step , Harper & Row, p. 300. ISBN 0-14-021978-1. 

De Bono, E., 1972, PO: A device for Successful Thinking , Simon & Schuster, New York. 

De Bono, E., 1990. Po: Beyond Yes and No, Penguin Books, London. 

Fellbaum, C. (Ed.), 1998. WordNet: An Electronic Lexical Database. Cambridge, MA: MIT Press 


141 



ISBN: 978-989-8533-23-4 ©2014IADIS 


Lehman, J, and Stanley, K. O.,2008. Exploiting Open-Endedness to Solve Problems Through the Search for Novelty. 

Proceedings of the Eleventh International Conference on Artificial Life (ALIFE XI). Cambridge, MA: MIT Press 
Lehman, J, and Stanley, K. O., 2012. Beyond Open-endedness: Quantifying Impressiveness. Artificial Life, vol. 13, pp. 
75-82 

Limb, CJ and Braun, AR, 2008. Neural Substrates of Spontaneous Musical Performance: An fMRI Study of Jazz 
Improvisation. PLoS ONEW ol. 3, No. 2: el679. doi:10.1371/joumal.pone. 0001679 
Lubart, T., 2005. How can computers be partners in the creative process: Classification and commentary on the Special 
Issue. International Journal of Human-Computer Studies - Special issue: Computer support for creativity, Vol. 63 
No. 4-5, Academic Press, USA. 

Maher, M.L. et al, 2013. Computational Models of Surprise in Evaluating Creative Design. Proceedings of the Fourth 
International Conference on Computational Creativity 2013 
Nachmanovitch, S., 1990, Free play: Improvisation in Life and art, Tarcher Penguin, USA. 

Snowball, http://snowball.tartarus.org/, [June 2014] 


142 



1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


AN EVS CLICKER BASED HYBRID ASSESSMENT TO 
ENGAGE STUDENTS WITH MARKING CRITERIA 
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ABSTRACT 

Over 4 iterations of a large course (>180 students) in introductory emedia design in a first year computer science course 
we have seen a year on year improvement. We believe this is down to the use of evs clickers for feed-forward 
assessment: that is to say a method of getting the whole class to evaluate previous cohorts’ submissions in public and 
discussing them, bringing to light the various properties they posess and how this maps to the marking rubric. This 
impacts on the students practices as they attempt their assignment. Over time, the practice has become more refined, 
principally through a rewritten criteria sheet, better training samples, and finally the development of a hybrid in-class 
assessment: the swarmative assessment combining both formative and summative practices and relying on its visibly 
social nature for its transformative power. This involves (a) evaluating previous submissions (in a non-graded way) - 
allowing for the free exercise of discriminative judgment not measured against any “authoritative” standard, but also (b) 
answering a set of objective questions about the work being assessed (what techniques were used to realize various 
effects). It ensures full cohort coverage together with engagement with the marking criteria. 


KEYWORDS 

Feed-Forward, EVS, Peer Asessment 


1. INTRODUCTION 

In the preceding 4 academic years, on a first year introductory Computer Science BSc module in multimedia 
development, the authors have used live feed-forward exercises using EVS clickers. Feed-forward essentially 
means getting students to mark previous students work to better understand the assignment criteria. Usually 
this is done in small groups with discussion. In our case, has involved using a lecture theatte with around 
200 students together. In it we use two projectors one to display exemplars of previous students’ work to be 
marked, and another projector to display graphs and average values of scores input by students in the 
audience using EVS clickers as they mark according to the rubric. For each previous assignment, students are 
led through the marking rubric and asked to mark the work as if they were the lecturer. Each dimension of 
the work being assessed has 5 attainment descriptors - students have to choose a number from one to five. 
When everyone has voted, then the average score is shown to the room as well as a histogram of numbers of 
students voting for each attainment level. 

What they are evaluating is a Multimedia CV assignment. This is a vehicle for students to be tested on 
their ability to design an interface and to be able to manipulate media to ensure the smallest possible file size. 
This is to test a learning outcome of learning how computers digitize and store media and the various fdesize 
reduction techniques available. Students are asked to create 5 screens in their CV: a home page with an 
animation on it, and four other pages for work, hobbies, education and contact details. It is therefore a 
certainly small assignment, but one which does require some planning, some sense of structure (particularly 
for the implementation of the audio player which needed to be independent of the other navigational actions 
of the user, i.e moving between screens) a house style, very small file size, and finally designed in such a way 
as to permit rapid resizing in the one hour final adjustments section before submission. 

This technique (of getting students to mark previous “exemplar” work) has been done before, but not in 
live public sessions. Notable examples are O’Donovan, Price and Rust [2004], Hendry, ArmsUong and 
Bromberger [2011] and Wimshurst and Manning [2012]. The theoretical basis for this kind of procedure 
comes most clearly from D Royce Sadler [1987] who beautifully expresses how just a few concrete 
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instantiations of seeing the marking criteria used in practice can make the whole experience of understanding 
and purposefully striving to achieve learning outcomes so much more comprehensible for students. 

This area of research has much in common with the more researched area of student peer-assessment (see 
particularly the meta studies conducted by Topping [1998], Falchikov and Goldfinch [2000], Zundert et 
al[2010] and much of the claimed benefits are in common (particularly in the area of meta -cognition). 
However, feed-forward, because it does not impact on the actual grades of the student whose work is being 
assessed (because they did it for a previous cohort) it may be more effective at improving meta-cognition in 
the assessor. The major practical difference between the work described in the papers above and ours is its 
public nature and immediacy. 


2. LIVE FEEDFORWARD AND STUDENTS’ REACTIONS TO IT 

In order to achieve the engagement of the entire cohort, each year we run this evaluation event twice: the first 
time formatively - looking at work done by previous cohorts, marking them and discuss the marking. The 
second time it is run summatively (credit bearing) - where students received a mark according to the level of 
agreement between their grading and that of the tutors (though this is marked extremely generously). 
However, this caused hostility in the first and third year it was used. Therefore in the last occasion we created 
hybrid assessment type - which we shall call the swarmative assessment - involving a formative part 
evaluating previous students work (which was not graded) followed by a small 10 question test on some of 
the practical features and techniques used in those previous artifacts which could be marked on an objective 
basis. It is there both summative and formative, however we argue that the power of this technique of live- 
feedforward occurs in the way it causes the student to compare the kinds of marks they are awarding with 
those given by the rest of the class. 

2.1 E-Media Design Historically 

The six below represent the scores for a large final assignment on the course. The x axis represents the final 
student percentage score in bands of 10%. The y axis represents the percent of the cohort in each decile. In 
this list, only students scoring above zero have been included. Although the first intervention using live 
feedforward occurs in 2010-11, we have included the two previous years in order to demonstrate that the 
results obtained in 2009-10 were not merely the result of the vastly larger cohort in that particular year. 
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As can be seen, a significant improvement takes place not only in the indicators of average grade, but also 
in tenns of the shape of the distribution. Tin 2009-10, the distribution becomes more normal, and shows a 
small overall improvement in average grade. 2010-11 was the first time we used the feed-forward method 
and did so with the identical criteria used in the 2009-10 assessment. This small improvement however came 
at the cost of some student hostility owing to the manner in which the feed-forward was conducted: namely, 
students were given marks by how near their marking pattern coincided with that of the tutor. Students 
complained of being asked to guess what we the tutors were thinking. 
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In 2011-12 therefore we rewrote the criteria to remove the subjectivism from the more basic things being 
tested (for instance “has the required number of screens”) and used likert scales for the more higher order 
criteria (appropriateness of design for instance). In 2012-13 the core course material for learning Adobe 
Flash was changed since the previous material had become outdated, though the assignment and rubric 
remained the same. 

In 2013-14 with the rubric only marginally changed, the modality of undertaking the feed-forward was 
changed. Just as in previous cases there would be a practice session and then there would be a marked 
second session - however, the marked second session involved both a formative assessment of the works 
being evaluated, as well as a small summative test on the objective properties of those works (for instance 
what technique allowed a certain effect to appear). Additionally, in response to the criticism of the quality of 
the training set, we introduced a new much better training set - producing past CVs but anonymizing the 
students’ personal details, but also putting pictures of another person of the same age, gender and ethnicity in 
their stead. This was meant to answer the concerns raised both about the training set and also about the 
method used for crediting participation in the feed-forward events. 

Though there are a number of variables in play, it does seem that quality of the overall student work has 
increased over that time, as well as the fact that it seems to have advantaged the entire cohort, and we believe 
that this is in large part due to the use of the feed-forward techniques. At this point, the question becomes: 
what is it that students are doing differently in their work to achieve this improvement? 

The principal evidence we will look at will be from two focus groups undertaken with the 2012-13 
(Em2012) cohort and the 2013-14 (Em2013) cohort where students from the course were asked to discuss 
their experience on it. Some secondary informing evidence will also be used from focus groups used on 
other courses: a humanities course on Web Publishing from 2012-13 (Hum2012), a joint focus group of the 
two 2012-13 courses together as well as one technology and engineering course (EmAHum2012), and a 
focus group from a 2011-12 Masters Course in Multimedia Computing (MmSpec2011). While the 
modalities of full cohort evaluation sessions were different on the other courses (Humanities involved a small 
class size with live evaluation primarily being used as a prompt for discussion) and the Multimedia masters 
course being used for full peer evaluation, nonetheless some of the feelings expressed by participants in those 
courses are generalizable to the participants in E-Media Design. 


3. REGULATING STUDENT EFFORT 

On the premise that students achieving higher grades are either doing more work or doing more efficient and 
goal focused work we sought to find out how students regulated their effort of the course of an assignment. 
This course is also interesting since it takes place in the first semester of their university career when they are 
making the transition to university study. 

In this light, how do students get a sense of what they are aiming for? Both focus groups mentioned the 
evaluation session as being crucial to getting a sense of the level required. The primary benefit was the clarity 
of the criteria. One student said: 

“definitely appreciated the seeing all the CVs - actually that was something I thought of when we 
were talking about the differences with going to university. . .saying I want x y and z, also there’s a 
proven example of what the finished product might look like, and what you thought of these 
different finished products that you've currently got up - an assignment where it's like, it seems like 
it’s clear what we are supposed to do, but all of us are going. I've no idea, and we don't see a 
finished product, so seeing the [examples], was really good.” (Em2012) 

This backs up what Royce Sadler has said about the value of concretizing abstract attainment criteria in 
specific exemplars which embody them. But while his research has typically focused on the understanding of 
higher-order criteria, the benefits also help with the satisfaction of the lower order criteria. Particularly, it 
taught students what to avoid 

“I think we could have all just looked at the CVs and gone that is a good one, that is a bad one but 
when you guys went through it and explained why that is a good one and why that is a bad one and 
pointed out the bad things and the good things then everyone got a better idea of what you expected 
of us. So like you said, the basic problems didn’t come up because you guys told us what the basic 
problems were so we all avoided them” (Em2012) 
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In 2012 one student said 

“I took like things from it and I saw things not what to do there were some things like maybe the 
sound was not appropriate or - like the buttons and the colour contrast that just clashes I had to 
guidance from that but it was good” (Em2012) 

As well as being able to avoid simple errors - it also helped with impasse management. Having a clear 
sense of what they are trying to achieve, seemed to help them identify points at which they were unable to get 
to their goal - and how to get round it (usually by recourse to help from fellow students). 

A student from the EM2013 focus group said: 

“My friend asked me if I could help her on the sound because they couldn’t get the sound buttons to 
pick up the music and I had forgotten how to make music like package sorter.” (Em2013) 

In the previous year, another student took a long term approach to the development of her project: 

“I looked at the Mark criteria and I thought, I am going to do what I can and if there are any extra 
bits I can do. I will try and do it then maybe I can try and ask for help. Because didn't we have the 
holiday during that period or something and when we come back from the holiday then I will ask at 
see if anyone can help me with the extra bits that I find difficult.” (Em2012) 

In other cases, students mentioned collaborating with fellows or with partners. In the 2013-14 group one 
student mentioned a facebook group of approximately 4 people in which he participated - in the 2012-13 
group a student talked of working together with others in the LRC (Learning Resources Centre) for bouncing 
off ideas about the most appropriate sound to use. He said 

“the most complicated bit was adding the sound. The CV itself didn't take that long about creating it 
everything was all right. But then me and my friends were at the LRC and we were all talking about 
how to put it in which one to use I think that was the worst thing because I would suggest one sound 
and then put it in the CV and it wouldn’t correspond with what I was talking about.” (Em2012) 

“Appropriateness” a term only used in the higher order criteria seemed to generate the most reflection. 
Another student from that year said 

“And choosing sound it took quite a while because I find something and then I’d question whether 
was really appropriate or not so then I’d have to look again and it was quite confusing what would 
be seen as appropriate” (Em2012) 

Another student talked about the major benefit of the marking exercises as being: 

“making sure and got all the buttons uniform all the same size all the text inside the buttons all the 
same size and all appropriate” (Em2012) 

Therefore the marking rubric seems to have done its job both at the lower end (causing students to avoid 
obvious pitfalls) but also at the upper end (causing sustained reflection, both individually and in groups about 
appropriateness and suitability). Also the clarity of the desired objective (obtained from seeing the exemplar 
works) has helped students metacognitively reflect on their means for achieving the goal. 

However, the rubric has only gone through two iterations - and the improvement of marks was already in 
evidence before the rewriting of the current rubric. Therefore what seems to have happened is that the ritual 
of the collective marking activity seems to have enhanced the authority and tangibility of the rubric. Lrom 
the 2009 date we can see there was a rubric - but it certainly wasn’t attended to by the students - regardless 
of the detailed description of how marks were arrived at, students seemed to have not attended to it. 
However, in the more recent iterations of the course, the rubric has come to be the corner stone around which 
their work revolved. One particularly pragmatic student said 

(Student)The sheet that you get basically gave us a list and markings of what you guys expected so it kind of told us 
exactly what to do. 

(Tutor) If it told you exactly what to do, while you were writing out your CV you were kind of ticking bits off? 

(Student)Well what I did was I read through all of it, then did my CV, then read through it all again and checked 
everything and compared it with my CV and then changed things around it. 

(Tutor) That is really interesting. Did you two do the same? 

(Student) Yes, pretty much. I just followed the requirements sheet. 

(Tutor) And did you do it formally as in ticking things off or did you do it. . . 

(Student)I used the (00.07.07) and come back to them eventually but I tend to do the easiest bit or the bits that I am 
not struggling with. 

(Tutor) So you did the easiest bits and sort of ticked them off and then you come back to the harder bit later. What 
were the harder bits? 
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(Student )The ones that involved more - making a background colour in grey is easy it just involves doing something 
in stuff but making things move around or something that involves (00.07.45) so you would come back to that. 

(Em2013) 

Here we clearly see the centrality of the rubric, its role in structuring work-flow and the attention to the 
easier and planning for the more complex, and in the earlier quotations, we saw how it also governed the 
occasions when students sought help from others. All of this is deeply similar to the participation-reification 
duality described in Wenger’s Community of Practice[1987]. The rubric represents a reification and 
codification of the practices necessary for a successful multimedia CV. The participation is the kind of work 
students do outside lectures and practices which make that successful multimedia CV - but also comprises 
the informal collaborations, help seeking and work planning which we have noticed above. However, as we 
have seen before, the mere fact of a rubric does not mean students will take notice of it, or structure their 
academic activity around it. But the faith in it, its authority and centrality comes from the ritual of the EVS 
public evaluation. Having learnt how to apply the rubric, by collectively marking 3-6 pieces of work 
together, the core centrality of the rubric becomes apparent to the students. 


4. THE SWARMATIVE ASSESSMENT 

A problem that has endured since first attempting this intervention is how to ensure compulsory participation 
in it. Were we to make participation a purely voluntary experience the danger is that the benefits would only 
extend to the attendees. In the first 3 iterations we ensured this by actually grading the students’ evaluation of 
previous exemplars - that was done by measuring the difference between the scores the tutors awarded and 
the scores the students awarded. In a focus group, one student said: 

I think the exercise is very good because it kind of gave you an idea of what was expected what you 
were looking for. The assessment of trying to determine what you would have marked them on, came 
across a bit harsh. The fact that we were being assessed on, speculating on. on what we think you had 
marked it on. So it was good in the sense that it gave you an idea of what to watch out for when you are 
doing your own... but because you were trying to assess on what someone else thought about somebody 
else's work, it was like a double stage hop (Em2012) 

In order to resolve this problem - requiring compulsory participation but without grading according to 
marking co-incidence we designed a hybrid assessment. This required the students to attend a session - the 
first part of which would involve the evaluations of a previous cohort’s artifacts which was not grade bearing. 
However, the second half would involve answering 10 objective questions about those artifacts (which tool 
was used to achieve effect x) which would be grade bearing - this meant it required everyone to be present - 
but the grade bearing part was a truly objective test and based on coincidence of subjective judgment. 

We justify the use of the word “swarm” because powerful effects can happen when students mark in 
public. They can compare the mark they awarded with the marks given by the cohort. One student said: 

for example you see websites and you and to you it's great and you confidently put excellent and when 
the rating comes out everybody says it was rubbish and wondering why did I? how did that come about? 
And so it challenges your intellectual you know thinking (Hum2012) 


5. CONCLUSION 

Chickering and Gamson [1987] choose “communicates high expectations;” as one of the seven principles for 
good practice in undergraduate education. The use of exemplars has been reported in a number of studies as 
being and effective way to achieve this. However, this typically takes place in small group situation involving 
discussion between tutors and students and is labor and resource intensive. We believe the technique we have 
shown here, offers a way of efficiently transmitting these expectations a very large cohort. Having an 
evaluation event involving both the subjective evaluation of exemplars as well as some objective test on the 
properties or techniques used in those exemplars appears the best way to satisfy the goal of full cohort 
engagement with assignment marking criteria. 
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ABSTRACT 

Recommender Systems (RS) have been applied in the Technology enhanced Learning (TeL) field for facilitating, among 
others. Learning Object (LO) selection and retrieval. Most of the existing approaches, however, aim at accommodating 
the needs of learners and teacher-oriented RS are still an under-investigated field. Moreover, the systems that focus on 
teachers, do not explicitly exploit their ICT competence profiles when providing LO recommendations. This is a 
significant drawback, since it can result in LO recommendations that are beyond the teachers’ competence to use. 
Towards tackling this issue, this paper extends previous work and proposes a teacher ICT Competence -based RS that 
considers teachers’ ICT Competence Profiles when recommending Learning Objects. Moreover, the results of its 
accuracy evaluation are presented. The results indicate that the proposed approach provides high predictive accuracy and 
outperforms commonly used, existing RS approaches. 


KEYWORDS 

Recommender Systems, Teacher ICT Competence, Learning Object Recommendation 


1. INTRODUCTION 

Recommender Systems (RS) are adaptive software applications aimed at providing suggestions for 
potentially useful items to the users (Bobadilla et al., 2013). They are widely used in many contexts, such as 
Movie Industry (Bobadilla et al. 2011) and Technology Enhanced Learning (TeL) (Manouselis et al. 2013a). 
RS are mainly divided in three categories in terms of the method applied for providing these 
recommendations (Ricci et al. 2011). These are (a) content-based recommenders, which build user models 
based on the items that users have already interacted with, (b) collaborative filtering recommenders, which 
produce recommendations for users based on the rating patterns of the users' neighbors (like-minded users), 
and (c) hybrid recommenders, which combine techniques from both previous approaches in order to reap the 
benefits of both, while tackling their individual drawbacks. 

In the context of TeL, one of the key applications of RS is the location and selection of Learning Objects 
(LO) (Manouselis et al. 2013b). The majority of existing TeL RS, however, is focused on the target group of 
learners and aims to accommodate their specific characteristics in the recommendation process, such as their 
learning styles or their learning preferences (Lerreira-Satler et al. 2012; Chrysafiadi & Virvou 2013). Lrom 
the standpoint of teachers, however, less research has been performed and few approaches actually consider 
teacher characteristics towards providing more personalized recommendations for their course design and 
delivery (Sergis et al., 2014). Due to the diversity in these characteristics, the "appropriateness" and 
usefulness of each LO to-be-recommended is different for each teacher based on her unique profile and 
needs. A significant type of such characteristics are the teachers' ICT Competences, since the latter are an 
important factor affecting their everyday practice (Sang et al. 2010; Goktas et al. 2013). Therefore, 
accommodating for these teacher attributes has the potential to provide significant added value in the teacher- 
oriented RS . 

Under this light, previous work was performed towards tackling the identified research problem of 
facilitating LO recommendation for teachers based on their ICT Competence profiles (ICT-CP). This work 
included the design and evaluation of a set of mapping rules linking teachers' ICT Competences to LO 
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metadata elements for identifying appropriate LOs for each teacher, competence-wise (Sergis et al., 2014). 
The contribution of the present paper is to extend this work by designing and developing an ICT 
Competence-based RS, which utilizes the proposed mapping rules and delivers informed LO 
recommendations to teachers based on their ICT-CP. Moreover, the predictive accuracy of the proposed 
approach is benchmarked against commonly used user- and item-based collaborative filtering approaches. 

The remainder of this paper is structured as follows. In Section 2 the current state of the TeL teacher- 
oriented RS landscape is presented. Moreover, the research problem and the contribution of the present paper 
is clearly identified. Section 3 presents the proposed approach for providing ICT Competence-based Learning 
Object recommendations to teachers. Section 4 comprises the evaluation method and results for validating 
the proposed approach's predictive accuracy. Finally, Section 5 contains the conclusions drawn and the future 
work in this research agenda. 


2. PREVIOUS WORK: TEACHER-ORIENTED TEL RECOMMENDER 
SYSTEMS 

In previous work (Sergis et al., 2014), a literature review was performed in the landscape of TeL 
Recommender Systems in order to highlight approaches that were addressed to teachers. The results of this 
process identified a set of 22 teacher-oriented RS. Moreover, an additional refinement of this list highlighted 
those approaches that created profiles with personal characteristics of the teachers (in addition to simple 
social data profiles) and utilized these data in order to drive more informed recommendations. This list 
comprises 7 RS and is presented in Table 1. Additional information depicted in Table 1 include the type of 
profile data that these RS capture. 

Table 1. Existing teacher-oriented TeL Recommender Systems 

# Paper Profile 

data 

Demographic data 

User work-related knowledge levels - not restricted 
to teachers 
Not clearly stated 

User interest on items based on social data and item 
descriptions 

Users experiences of resource use - not restricted to 
teachers 

Not clearly stated 

Demographic data. Information Technology 
experience - Low metric granularity 
As Table 1 depicts, each of the seven identified RS utilized a different type of user data in order to create 
their profile, while some did not include explicit information on the manner in which the reported profiles 
were created. However, a commonly identified issue in all approaches was the universal lack of consideration 
for exploiting the teachers' ICT competences. Bearing in mind the significance of these characteristics to the 
level of the level of ICT uptake of teachers (Vanderlinde et al. 2014), their integration in the LO 
recommendation process was proposed as a significant research problem. 

More specifically, a research problem was formulated, namely how can the process of LO 
recommendations to teachers based on their ICT-CP be facilitated. This is reinforced by the reported need for 
more informed user modeling processes towards better recommendations (Adomavicius & Tuzhilin 2005). A 
first step towards tackling this issue was performed by constructing and evaluating a set of mapping rules 
linking (a) specific ICT Competences as they are depicted in the UNESCO ICT Competency Framework for 
Teachers (http://tinyurl.com/ocaxgqk) (ICT-CFT) to (b) specific Learning Object Metadata attributes, as they 
were depicted in the OpenDiscoverySpace (ODS) Project (http://www.opendiscoveryspace.eu) LOM 
Application Profile (Sergis et al., 2014). This work showed that the proposed mapping rules provided high 
accuracy and could, therefore, effectively classify LOs as appropriate in terms of each teacher's ICT-CP. 


1 Bozo et al. (2010) 

2 Schoefegger et al. (2010) 

3 Sielis et al. (2012) 

4 Schirru et al. (2010) 

5 Aehnelt et al. (2008) 

6 Cobos et al. (2013) 

7 Zapata et al. (2013) 
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The next step in this research agenda is the investigation of the potential added value that this approach 
could deliver in the teacher-oriented RS. Therefore, the aim and contribution of the present paper is to 
progress this research agenda by reporting on the process of designing, implementing and evaluating a RS 
that utilizes these mapping rules in order to deliver ICT Competence -based LO recommendations to teachers. 
More specifically, the RS overview is presented and its recommendation methodology is described in detail. 
Finally, the predictive accuracy evaluation of the system is presented and the added value of this approach is 
discussed. 


3. ICT COMPETENCE-BASED LEARNING OBJECT RECOMMENDER 
SYSTEM FOR TEACHERS 


The overview of the proposed ICT Competence-based approach to recommending LOs to teachers is 
presented in Figure 1 . 



Active 

User 


Initial Learning 
Object Pool 


^ICT CP 
|l Profile 


Mapping 

Rules 


a%a 


ICT 


Refined Learning 
Object Pool 


Competence 

-based 

Neighbors 


LO Recommendations 


V Only ICT Competence-based "Appropriate" Learning Objects 
are considered 

V Neighbors' ICT Competence Profile similarity as weighting 
factor 

V Neighbors' ratings of LO to be recommended 


Figure 1. Overview of proposed teacher ICT Competence-based Learning Object Recommendation System 


The methodology for creating LO recommendations includes an initial pre-filtering of the full set of LOs, 
in order to keep only those that correspond to the teacher's ICT-CP. This step is realized by utilizing the 
mapping rules designed and evaluated in previous work (Sergis et al., 2014). At the end of this process 
(which is performed offline), a refined pool of LOs has been created that contains the "appropriate" LOs for 
the active teacher. Following this step, this refined pool of LOs is used for delivering the recommendations 
based on the ICT Competence -based neighbors of the active user. Therefore, the proposed approach utilizes 
the teachers' ICT-CP in a twofold manner, namely (a) as a pre-filtering mechanism for identifying LOs that 
are beyond the active teachers' competence to use, and (b) as an alternative neighborhood creation method, 
focusing on ICT competence profile similarities instead of common rating patterns. 


152 



1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


As the Figure 1 describes, the proposed approach comprises a set of distinct steps, each of which are 
presented in the following sections. 


3.1 Teacher ICT Competence Profile Depiction 

As aforementioned, the teacher ICT-CP was depicted using the UNESCO ICT Competency Framework for 
Teachers (CFT). The reasons for selecting the CFT over alternative frameworks have been discussed in 
Zervas et al. (2014). This framework addresses all aspects of a teacher’s practice within their work context 
and aims to provide comprehensive teacher ICT competency standards for assisting them in successfully 
addressing all relevant dimensions. It comprises 6 categories of ICT Competences spanning 3 proficiency 
levels. Each competence category is further divided in sub-competences that are differentiated according to 
the proficiency levels. The total number of sub-competences described in the CFT is 61. Therefore, a teacher 
ICT Competence Profile submitted to the system would include each of these sub-competences, evaluated in 
terms of whether the active teacher possesses it or not. 

3.2 ICT Competence-based Neighborhood Creation 

The neighborhood creation process in the proposed approach is performed in a novel manner. More 
specifically, the method employed in the proposed approach utilizes the teachers' ICT-CP in order to create 
these neighborhoods. This alternative has the potential to provide more informed recommendations, since 
users (in our case, teachers) would receive LO recommendations based on the opinions of colleagues that 
have the same competence background with them. 

In the context of this paper, the ICT Competence-based neighborhoods were constructed using the 
Jaccard co-efficient (JC) (Verbert et al. 2011). This coefficient measures the similarity between two sets (in 
our case ICT Competences) by dividing the number of common items with the number of different items 
from both sets of rated items. It is calculated using the formula: 

lf g n fpl 

1 ~ l/t,l+ U v \- l/[, n / v l 

where ly and /y-are the Competences of users U and V respectively. 

Within the context of the proposed RS, the output of this formula served as an indicator of the "closeness" 
between two users, i.e., how similar are their ICT Competence profiles. This indicator was utilized in the 
recommendation process of the proposed approach (section 3.4) as a weighting factor for predicting the 
rating of the active user based on the ratings of her neighbors. 

3.3 Pre-filtering Learning Objects 

The second step in the proposed ICT Competence -based recommendation system relates to the pre-filtering 
of the available LOs in order to consider only those that are "appropriate" for the active teacher, in terms of 
her ICT-CP. This process was performed by utilizing the set of mapping rules linking specific ICT 
Competences described in the CFT and specific LO metadata elements. Therefore, for each LO the 
corresponding mapping rule was fired and the system would flag the LO appropriately in terms of the active 
teacher's ICT-CP. At the end of this process (which was performed offline to reduce the systems' response 
time), each teacher would have their own unique set of "appropriate" LOs, based on their ICT-CP. Finally, 
utilizing this refined pool, the system would deliver the LO recommendations. 

3.4 ICT Competence-based Learning Object Recommendations 

The process of delivering the LO recommendations is similar to the existing collaborative filtering 
approaches. More specifically, a weighted average of the neighbors' ratings was utilized in order to predict 
the rating that each user would assign to a LO that was to-be-recommended. The difference with existing 
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collaborative filtering approaches, apart from the fact that the candidate LOs would come from the refined 
pool of "appropriate" LOs instead of the full pool of all available LOs of the repository, is the nature of the 
weights assigned to the weighted average metric. More specifically, as aforementioned in a previous section, 
these weights were the output values of the Jaccard coefficient used for creating the competence -based 
teacher neighborhoods. Therefore, "closer" neighbors, i.e., neighbors with more similar ICT-CPs to the active 
user and higher Jaccard coefficient, had a greater impact in the recommendation process. 

The predicted rating r U i for each LO i by the active user u was calculated based on the formula: 

El*} **Ji 

r “ ' i 

where r., is the rating provided by neighbor j on the LO i, j is the jg, neighbor of the active user and N is 
the neighborhood size utilized. Based on the generated predicted rating, the system could infer not only if 
each LO would be appropriate for the active user, but, moreover, whether it would be of interest to her. 

The proposed system was evaluated in terms of its predictive accuracy. The evaluation methodology 
followed is described in the following section. 


4. EVALUATION 


4.1 Dataset 

An experiment was performed towards the evaluation of the proposed recommendation method based on 
teachers' ICT Competence profiles. The dataset which was utilized came from the OpenDiscoverySpace 
(ODS) Project (http://www.opendiscoveryspace.eu). The reason for selecting this dataset was that it 
contained existing ICT Competence profiles provided by 115 real teachers. The overview of the retrieved 
dataset is presented in Table 2. Aggregation Level 1 LOs refer to standalone Educational Resources (e.g., 
flash simulations), while Aggregation Level 2 LOs refer to Lesson Plans. 


Table 2. ODS Dataset Overview 


Category 

Sample Size (N) 

Rating Data 

User ICT-CP 

115 

- 

AL 1 Learning Objects 

523 

1375 

AL 2 Learning Objects 

475 

986 


4.2 Evaluation Method 


The evaluation was based on the predictive accuracy of the proposed approach. It involved the standard 
"Leave-N-out" technique, where 70% of the data are used as a "training" set for the recommender system and 
the remaining 30% of the data are used as a "test" set for evaluating the performance of the system; in our 
case, its predictive accuracy. The Root Mean Squared Error (RMSE) metric was selected for measuring the 
predictive accuracy of the system (Shani & Gunawardana 2011). This metric is calculated based on the 
formula: 


RMSE = 



where is the set of generated predicted ratings, »" U i is the set of known ratings and T is the set of users 
and items for which the ratings are known. 

Moreover, the proposed approach was benchmarked against two "control" recommendation methods, 
namely a user-based collaborative filtering approach and an item-based collaborative filtering approach. Both 
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these approaches utilized the Pearson correlation coefficient ( PCC) for creating the user/item neighborhoods 
( Verbert et al. 2011). The Pearson correlation coefficient is calculated based on the formula: 

PCC — ~ ~ ^ 

♦'u ) 2 

where / is the set of items that both users u and w 7 have rated, and denote the ratings of the users u 
and w on the item i a while r u and % denote the average ratings of the two users respectively. Following this 
step, both "control" approaches utilized the PCC as a weighting factor for creating the predictions of the 
active user for the LOs that they had not yet rated. 

Finally, the evaluation experiment was run for an increasing number of neighborhood size in order to 
monitor the behavior of each approach in each occasion. The results of the evaluation experiment are 
presented in the following section. 

4.3 Evaluation Results 


The results of the evaluation experiment are presented in Figure 2 and Figure 3. The former presents the 
results concerning Aggregation Level 1 LOs, while the latter presents the results for Aggregation Level 2 
LOs. The reason for examining both these LO Levels is that in the "Pre-filtering" step of the proposed 
approach, different mapping rules are utilized for these LO Levels, and therefore, both had to be evaluated 
compared to the existing approaches. 



Neighborhood Size 


Figure 2. RMSE results for Aggregation Level 1 Learning Objects 

As the Figures 2 and 3 depict, the results of the evaluation experiment are very promising. More 
specifically, the proposed approach achieved high levels of predictive accuracy (in the form of low RMSE 
value) for both Aggregation Level LOs. This means that it can effectively identify LOs as appropriate or not 
in terms of the teachers' ICT Competences, as well as to select those that are interesting to her. 

Moreover, compared to the user- and item-based approaches, a significant improvement was detected. 
This means that incorporating the teachers' ICT-CP in the LO recommendation process (and the manner in 
which this was performed) can provide added value in existing RS systems and offer teachers with more 
useful services. 

Another important finding is that ICT Competence -based recommendations out-perform existing 
commonly used collaborative filtering approaches for all neighborhood sizes, signifying that they could 
prove beneficial in tackling some common collaborative filtering problems, such as the new user problem 
(Bobadilla et al. 2012). 
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Neighborhood Size 


-User-based PC C 
Item-based PCC 
■ Competence-based JC 


Figure 3. RMSE Results for Aggregation Level 2 Learning Objects 

The new user problem occurs within RS when it is not possible to make reliable personalized 
recommendations due to an initial lack of ratings by the active user. This is more evident in educational 
datasets, which are usually very sparse. The results of this paper show that the inclusion of the teacher profile 
information as a means for creating neighborhoods could assist in remedying for this issue in a significant 
degree, since the active user could be linked to a set of appropriate neighbors without the need for their own 
ratings. 

The above overall findings provide with strong evidence that the inclusion of teachers' ICT-CP in the 
learning object recommendation process within Learning Object Repositories (LORs) can provide enhanced 
personalized services in a number of ways and can accommodate each teachers' individual profile and 
competence-related needs. 


5. CONCLUSIONS AND FUTURE WORK 

This paper, building on previous work, proposed a system that integrated the teachers' ICT-CP in the LO 
recommendation process within LORs. This work was focused at tackling a research gap in the RS 
landscape, related to the lack of consideration of the teachers' ICT-CP when recommending LOs. This was 
considered as a significant drawback, bearing in mind the apparent importance of these personal 
characteristics in the everyday level of ICT uptake of teachers. 

Towards measuring the level of the added value of the proposed approach, an evaluation experiment was 
performed, which focused on benchmarking the predictive accuracy of the approach against two commonly 
used collaborative filtering approaches. The results provided with evidence that the proposed approach can 
deliver better recommendations (in terms of low system RMSE) for all LO Aggregation Levels and can also 
perform adequately in small neighborhood sizes. 

Bearing in mind the added value that the proposed approach showed to deliver in the LO recommendation 
process for teachers, future work should be focused on identifying a robust framework for eliciting these 
teacher characteristic from their usage patterns within LORs. The need for this springs from the fact that 
teachers (and users in general) are usually either unwilling to provide personal information or when they do, 
the validity of the provided data cannot be ensured (Belk et al. 2013). Therefore, in order to overcome the 
above issue, and reap the reported benefits of ICT Competence -based LO recommendations, mechanisms 
should exist for inferring the teachers' profile from their usage patterns (Marin et al. 2014). 
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ABSTRACT 

Reading in the content areas of science, social studies, and current events is a difficult task that is even more elusive to 
Spanish speaking English language learners. There is a huge increase in children transitioning from their LI (e.g., 
Spanish) to L2 (e.g., English) in classrooms across the US. These ELs face challenges due to a lack of fluency in 
decoding, vocabulary, and word, sentence, and discourse level complexities in English learning. Structure strategy 
instruction on the Web for English Language Learners (SWELL) is a web-based tutoring system that supports ELs in 
reading comprehension by teaching them about five text structures. In addition SWELL provides two adaptations for ELs 
- Spanish Scaffolding (where students were presented materials in both Spanish and English) and English Hybrid (where 
students were given the option of seeking assistance in Spanish by hovering over words, clicking on sentences, or 
viewing a full page in Spanish). In this paper we report on the design and pilot studies conducted within five classrooms 
at grades 4,5, and 6. Our results show improvements in reading comprehension measured by researcher design measures. 


KEYWORDS 

Web-based intelligent tutoring, reading comprehension, computer based learning 


1. INTRODUCTION 

Reading in the content areas of science, social studies, and current events requires excellent decoding, 
vocabulary, oral language, and comprehension skills. Unfortunately most high-stakes assessments of reading 
comprehension show poor results for over 33% of children in grades 4 and 8. The numbers are even worse 
for Spanish-speaking Latino children who are most at risk for school failure. In 2008, the U.S. high school 
dropout rate for Latino adolescents was 22.5% compared with all other sub groups combined (Fry & 
Gonzalez, 2008). English language learners are at particular high risk for poor educational outcomes due to a 
myriad of factors including poorer reading performance (Proctor, Carlo, August, & Snow, 2005). They are 
most likely to drop out of school and have a lower high school graduation rate than any other group. Spanish 
speaking ELs also perform poorly on reading comprehension and mathematics standardized high stakes 
assessments like the National Assessment of Educational Progress (2013). 

Researchers have suggested that improving vocabulary skills and knowledge of comprehension strategies 
can improve reading comprehension outcomes for ELs (Baker & Dalton, 2011; Proctor, Dalton, Grisham, 
2007; (Jimenez, 1997; Jimenez, Garcia, & Pearson. 1996; Proctor et al., 2005; Proctor et al., 2011). 

The SWELL project is an outgrowth of a successful web-based intelligent tutoring for the structure 
strategy (ITSS) project where we developed and tested the system to teach 4 th through 8 th grade children the 
structure strategy. The system showed statistically significant differences favoring the ITSS groups in large 
scale randomized controlled trials in grades 4-8 (Wijekumar, Meyer, & Lei, 2012, 2013; Wijekumar& 
Meyer, 2012; Wijekumar et al., in press). The goal of the SWELL project was to extend the ITSS system by 
presenting two specific adaptations for Spanish speaking ELs - Spanish Scaffolding or English Hybrid. 
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We present our findings from the development and pilot studies here. Our findings support the growing 
research evidence base in support of the text structure strategy and web-based delivery of the instructional 
materials. Finally, this is the first development where the structure strategy has been used with Spanish 
speaking ELs in grades 4, 5, and 6 and showed promise in improving reading comprehension. 


2. SWELL AND ITSS 

SWELL (Structure strategy instruction on the Web for English Language Learners) is an extension of the 
ITSS system. An animated pedagogical agent named I.T. appears on the screen and presents modeling, 
practice, assessment, scaffolding, and feedback to learners. Each student is assigned a unique username and 
password which they use to log into SWELL and proceed at their own pace. Students learn how to find 
signaling words used by authors (e.g., in contrast) Figure 1, classify the text structure (e.g., comparison), 
summarize the text using a main idea pattern (e.g., dogs and cats were compared on friendliness, diet, and 
exercise patterns), and recall the text. 

Instruction is presented about five text structures - comparison, problem and solution, cause and effect, 
sequence, and description. Text structures are presented in the sequence listed here. Approximately 12 
lessons are available for each text structure. Lessons are presented from various domains (e.g., science text 
followed by sports). Passages are appropriate for each grade level learners. 



Q ITSS - Intelligent Tut 
f C 2 itss.psu.edu/I 


Figure 1. SWELL Lesson to find signaling words 
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2.1 Theoretical Foundation of the Structure Strategy 

The overall theoretical rationale for the development of SWELL shown in figure 1, rests on the cognitive 
schemas developed through the use of the structure strategy to improve reading comprehension (Meyer, 
1975). During this training EL learners will learn how to identify the signaling words in the expository text 
(e.g., different, in contrast are signals for the comparison text structure), classify the signaling words into the 

text structure, write a main idea for that text structure (e.g., and were compared on , , and 

), and use both the main idea and the signaling words to write a full recall of the passage. 

Creating memory representations of the texts using the text structure, aids in the strategic organization, 
associations, and ease of recall, resulting in improved reading comprehension. The quality of the memory 
representations are gauged based on student written protocols that are scored based on the use of top-level 
structure to organize recall, use of signaling words, quality of the expression of the main idea, and details 
remembered. 

EL learners without the necessary prior knowledge and English language skills will require additional 
English language assistance in the form of contextual cues and vocabulary assistance in order to be able to 
receive the structure strategy instruction in a meaningful manner. 

2.2 SWELL Adaptations for Spanish Speaking ELs 

SWELL has seven unique features noteworthy for all learners but very relevant to the EL learners who are 
the focus of this project. These design features are part of the existing ITSS and are also used within 
SWELL. 

Lirst, SWELL narrates (with a human voice) all procedural information and instructions to the learner. 
SWELL also has narrations for the text passages and feedback given to the learner. Additionally, as I.T. reads 
a passage, the words blink on the screen allowing learners to follow the reading on the screen. This design 
mimics an adult reading to the child by pointing their finger on the words on the page as they read out aloud. 
This can assist EL learners who have difficulty processing the words. SWELL also has Spanish Language 
narrations for each procedure and passage. Animations are created to match the Spanish narrations. Lesson 
texts were carefully reviewed to meet the linguistic needs of ELs. 

Second, the SWELL system progressively gives more detailed hints to the learner depending on how 
many tries they have completed for the practice assignment. Lor example, if the student is on their third try 
and they have not mastered the concept, I.T. presents a pop-up window on the screen with very detailed 
helpful hints and read the hints to the learner. The number of tries for each lesson and question vary from two 
to eight. This is designed to reduce any gaming related to tries. Lor example, when students realize that the 
system will give them more help after the second try, they tend to wait for the second try to get the correct 
answer. By varying the number of tries for each question, students cannot rely on a pattern to game the 
system. 

Third, SWELL shows children how to use the structure strategy to read and comprehend texts in 
academic domains such as science, social studies, and current events. Lor example, in the first and second 
lesson students read a comparison text passage - differences between favorite Presidents - Lincoln and 
Washington. In the third lesson they read a science text comparing African and Asian Elephants. In the fourth 
lesson they read a passage comparing Olympic athletes, Dara Torres, Michelle Kwan, and Mary Lou Retton. 
This approach shows learners how the same text structures are applicable to all expository texts. To support 
the linguistic needs of ELs SWELL previews each lesson with vocabulary instruction and paraphrases of 
complex sentences. 

Lourth, SWELL shows children how real-life texts are organized to promote transfer to environments 
where explicit signaling may not be readily available. This approach is part of the scaffolding process that 
allows gradual fading of supports as the student moves through the lessons. 

Lifth, SWELL has alternative approaches to presenting information to the learners. Lor example, in the 
comparison of Olympic athletes, students click on a matrix showing how the each athlete is compared on the 
sport they played, the number gold medals won, and the year of their first Olympic win. In some cases a 
hierarchical diagram is shown to the learner focusing their attention to the logical organization of the text. 

Sixth, SWELL shows learners how to combine text structure. Lor example, after completing about 10 
lessons on the comparison text structure and another 10 lessons on the problem/solution text structure. 
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students are shown how they can combine comparison and problem/solution in one passage. Many real-life 
texts combine text structures and it is important that students learn how to read such texts for the structure 
strategy to be effective in improving their comprehension of academic texts. 

Finally, SWELL allows the teacher to access all the students’ responses at any time through a teacher 
viewing tool and provides bi-weekly reports to the teacher on student performance and gaming. The teacher 
can also modify the students’ pathway through the lessons if they see that they are able to understand the 
content. For example, the teacher can take place the student in a more advanced lesson if they believe that 
the student has the background knowledge to understand the content. 

SWELL Extension 1 - Spanish Scaffolding for each lesson . Students are provided narrated (and on- 
screen text) procedural information about each lesson in Spanish. E.g., for the earlier example on finding 
signaling words, "Cuandoyoleo el pasaje, buscolaspalabras de senalizacion. 6 Puedesver la palabra de 
senalizacion, diferentes en el pasaje?" After these instructions, students will read the passage in Spanish (with 
I.T. narrating the text in Spanish). Then I.T. gives the students a preview of what is expected in the English 
Language version of the lessons. The student is finally placed in the English lesson to apply the skills they 
learned in their LI. Figure 2 presents a lesson comparing two types of elephants with Spanish scaffolding. 




Elefantes 

Dos diferentes especies de elefantes existen hoy en 
dla; estos dos tipos son el elefante afncano y el elefante 
Indio. Estas interesantes creaturas difieren 
dramiticamente en orejas. lomos y en que tanto viven. 
Los elefantes africanos tienen orejas muy grandes. Sus 
lomos se arquean hacia abajo a la mitad. Los elefantes 
africanos viven de 50 a 60 ados. 

Los elefantes Indios tienen orejas pequehas. Los 
lomos de los elefantes indios se arquean hacia arriba a 
la mitad. Elios viven de 70 a 80 anos. 


Pm up Window 




Haga clic en las palabras claves: 

1 . 

2 . 


Figure 2. Spanish Scaffolding Lesson Screen 
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This adaptation is designed to take advantage of the Spanish literacy skills that students may have. It has 
been shown that LI literacy skills transfer very well to L2 as noted by Cummins’ dual language iceberg 
theory (Cummins, 1979). It has also been shown that tapping into their LI literacy can aid their 
understanding of L2 - hence the Spanish version of the passage (Slavin& Cheung, 2003). 

Two versions of the English language passage are available to the learners. The first form of the text is at 
grade level readability. The second is an easier version of the passage created using features identified by 
previous research studies (Abedi, 2006; Brown, 2007; Williams, et al., 2009). Students are initially placed 
into reading the easier versions of the English passage and gradually transitioned into the regular grade level 
lessons based on their scores in each lesson (e.g., if they are able to score 80% or higher on main idea tasks, 
they may be Uansitioned into the grade level passage in the next lesson). 

SWELL Extension 2 - English Hybrid for each lesson: Each SWELL lesson has an English preview 
lesson for the EL learners. Examples of elements covered in this preview are: vocabulary enhancements 
(activities designed to identify cognates, roots, affixes, and morphological relationships; using words in 
context, and learning synonyms) and previewing sentences in easier versions. In these adaptations students 
can also click on specific words on the screen and hear the word meaning in English. Students may highlight 
a whole sentence and have I.T. give them an alternative simplified sentence to help them understand it. The 
vocabulary enhancements will use current research by Proctor et al., Dalton et al., Kamil& Herbert, 2005, 
and Beck, McKeown, &Kucan, 2008. 

After completing these previews of the lesson, the student is allowed to complete the full lesson by 
clicking on signaling words, writing a main idea, and writing a full recall of the passage. 

This adaptation is designed specifically for learners who do not have the Spanish literacy or oral language 
proficiency necessary to first practice in their LI. Instead we are trying to give them scaffolds to develop 
their English language vocabulary and background knowledge necessary to comprehend the grade level texts. 
Again, students initially read the easier version of the English passage prepared for the Spanish Scaffolding 
enhancement. As they gain confidence and their performance meets the thresholds set after the iterative 
studies, they are Uansitioned into grade level text passages. 


3. RESULTS FROM PILOT STUDIES 

The research team has recently completed three series of quasi experimental research studies on the SWELL 
software with children in grades 4,5, and 6. Our findings on a single subject design study showed children 
using the SWELL software made expected progress in the signaling word, text sUucture classification, main 
idea, and recall tasks. Usability tests showed that children using the English Hybrid version of SWELL were 
unable to navigate the system without explicit insttuctions in Spanish. In addition, the usability studies also 
showed that most children in Southern Texas did not understand the accent of the narrator who was from a 
different part of the counUy but spoke Spanish. The team revised all the narrations using a local speaker to 
ensure that all children in the area would understand the narrations. Finally, the extended study with a 
matched control group showed that the SWELL classroom children outperformed the matched sample on 
signaling word and main idea tasks. 


4. CONCLUSION 

Based on the iterative design experiments conducted during years 1 and 2 we have made adjustments to the 
SWELL lessons, sequences, and types of support for the children. This research is the precursor to a full year 
randomized pilot study planned for this academic year. The preliminary research showed positive outcomes. 
The year 3 studies are under way and pre-testing has been completed. Students have started using the 
SWELL software in the classrooms and teachers are reporting that students are learning well. 
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ABSTRACT 

New standards require teachers to integrate the use of technology in their teaching and preparing teachers at the 
preservice level to integrate technology into the classroom is key. The way in which this is accomplished varies across 
institutions though often a technology tools course stands as an individual course with the hope professors are modeling 
the use of technology in their own teaching. In this paper we describe a process of integrating technology knowledge, 
pedagogical knowledge and content knowledge within a block one course in a special education teacher preparation 
program. In addition, we use a pre/post and follow-up survey to assess the change in perceptions and use of technology 
surrounding the course and at the end of the 5 -semester program upon completion of student teaching in real world 
classrooms. Results show teacher candidates report learning about educational technology tools and the integration into 
their teaching as important and necessary. Learning about and using educational technology integration had statistically 
significant positive impact on their teaching as well as enhancing the engagement of k-12 students. Specifically, 
proficiency with Smart board and Movie Maker was positively impacted by the knowledge and skills they developed 
from the course. The importance of collaboration between teacher preparation programs and field experiences in ensuring 
the appropriate use of educational technology for learning is emphasized. 


KEYWORDS 

Educational Technology, Special Education, Teacher Preparation. 


1. INTRODUCTION 

Successful technology integration into pedagogy can improve student learning (Wright & Wilson, 2012; 
Gupta & Fisher, 2012). Relevant studies concluded that using technology in educational settings benefits 
student learning in k-12 environments (Gulbahar, 2007; Kim & Hannafin, 2011 in Liu, S-H. 2011). Effective 
technology integration happens across the curriculum in ways that deepen and enhance the learning process. 
In particular, technology integration must support four key components of learning: active engagement, 
participation in groups, frequent interaction and feedback, and connection to content and real-world experts. 
Technology should be implemented in the classroom only if its role in a given instruction is determined along 
with pedagogical issues related to a given instructional task (Okojie et al, 2006). This also helps to define 
active learning. Specifically, active learning includes hands-on tasks and authentic activities that include 
experiential learning, collaborative learning, context-based learning, and computer-based learning. Authentic 
learning values learner-centeredness; the learning tasks revolve around learning experiences that are 
connected to real-world situations. Students should be encouraged to work with information to derive 
meaning and understanding, form new mental representations of the material, and construct and reconstruct 
new knowledge based on their experiences (Keengwe, & Georgina, 2013). Effective technology integration is 
achieved when the use of technology is routine and transparent and when technology supports curricular 
goals. 
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The move from traditional education courses and programs towards technologically enhanced traditional 
classrooms and pedagogies has been slow. The result of this slow movement seems to suggest that while low 
level use of technologically enhanced pedagogy is wide-spread, high-level use is more sporadic (Georgina & 
Olson, 2008). Even with the slow adoption recent research is finding that technology integration strategies 
can be associated with gamification and game-based learning practices in Math and Science (Luminea, 
2013). Preparing teachers at the preservice level to integrate technology into the classroom is needed to 
stimulate this change and must become an increasing focus for teacher education programs (Chai, Koh & 
Tsai, 2010). The way in which this is accomplished currently varies across institutions though often a 
technology tools course stands as an individual course with the hope professors are modeling the use of 
technology in their own teaching. Although new teachers report higher levels of comfort with technology; 
more experienced teachers report using technology more often in the classroom when delivering instruction 
or having students engage in learning activities (Russell, O’Dwyer, Bebell, & Tao, 2003). Findings such as 
this motivated the development of a course that integrated instruction of educational technologies with 
special education content: SPED 401. 

In this paper we describe a process of integrating technology knowledge, pedagogical knowledge and 
content knowledge within a block one course in a special education teacher preparation program. While our 
research addresses four questions described later, we focus on one in this paper: 1) What impact does 
participating in SPED 401 ( special education course that integrated educational technology ) have on teacher 
candidates’ perceptions of proficiency with hardware and software immediately following the course and two 
years later? We use a pre/post survey to assess the change in perceptions and use of technology surrounding 
the course as well as at the end of the 5-semester program after student teaching in real world classrooms. 
Keengwe and Georgina (2009) noted that simply using technology does nothing to enhance pedagogy; rather 
than viewing technology as merely a tool for delivery, it should be seen as a means to improve learning. 
Supporting this view our early results indicate teacher candidates report learning technology skills that both 
enhance presentation of material and facilitate student learning in the classroom. 

1.1 Technology Integration 

Technology integration is the implementation of technology during teaching. Technological Pedagogical and 
Content Knowledge (TPACK) model is used by teachers, instructional designers, and instructors to facilitate 
the determination of learning strategies which align and integrate technologies with content -based objectives. 
TPACK framework has five progressive stages which serves to guide teachers through integration practices. 
These five stages include recognizing technology, accepting technology, adapting lesson delivery and 
assessment methods, exploring and implementing technology, and advancing and reflecting on technology 
implementation and lesson content (Niess, Ronau, Shafer, Driskell, Harper Johnston, & Kersaint, 2009) 
According to Niess et al., the model is used for examining the interrelatedness of content knowledge, 
pedagogical knowledge, and technological knowledge (2009). The TPACK model approach encompasses 
many different philosophies and styles of teaching (Mishra & Koehler, 2009). The TPACK framework is 
usually illustrated by a Venn diagram which reveals the intersections of technological knowledge, content 
knowledge and pedagogical knowledge. It is the overlap of the three types of knowledge that allows for the 
efficacious integration of technology into pedagogy (Graham, Borup, & Smith, 2011). A transformative 
affect occurs, increasing the integration of technology into pedagogy, as teachers move from one TPACK 
level through the next (Puentedura, 2013). 

In our knowledge-based education system and professional society, 21 st century skills are essential (Trilling 
& Fadel, 2009). 

In SPED 401, special education teacher candidates learn technology skill-based approaches to lesson plan 
development. For example, these approaches include personal learning mobile devices like iPads, which 
accentuate collaboration and personalization in classrooms (Manuguerra & Petocz, 2011). Technology tools 
already enhance and support learning for special needs students (screen readers, magnify, speech to text...). 
TPACK framework outlines a process for selecting and integrating technology into specific content. 
Knowledge of this process is needed by special education teachers of the 21 il Century. 
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1.2 Teaching Technology Integration in Preservice Teacher Preparation 
Programs 

The National Council for Accreditation of Teacher Education (NCATE) developed the National Education 
Technology Standards for Teachers (2008); these standards require that teachers use technology in their 
classrooms, and design learning environments and experiences that support teaching, learning, and curricula 
(Liu, 2011). The standards have also led teacher education institutions to acknowledge the need for teacher 
preparation to ensure new teachers know how to use technology as an effective instructional tool. Liu (2011) 
and others indicate that teacher education institutes are natural places for training teachers how to integrate 
technology into daily classroom learning. Liu (2011) states that empirical evidence indicates that teacher 
education programs have not taught new teachers how to use technology effectively; that is, preservice 
teachers still lack the ability and knowledge needed to teach successfully with technology (Angeli, & 
Valanides, 2008 in Liu, 2011). This finding coincides with the U.S. Department of Education’s Blueprint for 
Success, which states the need for schools and districts to optimize the use of technology, recognizing 
educational Success, professional excellence, and collaborative teaching (2013). 

Chen, R-J (2010) found preservice teachers’ self-efficacy of teaching with technology had the strongest 
influence on technology use, which was mediated by their perceived value of teaching and learning with 
technology. School’s contextual factors had moderate influence on technology use. Moreover, the effect of 
preservice teachers’ training on student-centered technology use was mediated by both perceived value and 
self-efficacy of technology. It is essential for teachers to see the value of educational technology and build 
their competence to use technology effectively. Teacher educators need to reconsider their training 
approaches in order to cultivate positive attitudes towards education technology and develop preservice 
teachers' competence in using educational technology for teaching and learning. 

When preservice special education teachers are trained in technology skill development, there is a direct 
alignment with their delivery of content which impacts student learning outcomes (Eerreira, Baptista, & 
Arroio, 2013). Long (2013) suggests that the students should work directly with both the subject content and 
the technology tools and platforms in order to ensure a more comprehensive understanding of ideas, 
concepts, and practices. Teacher pedagogical beliefs are important when exploring technology integration. 
These beliefs play a critical role in successful technology integration (Ertmer, 2005; Hermans, et al., 2008; 
Tondeur, van Keer, van Braak, & Valcke, 2008). Beliefs about teaching can be called “preferred ways of 
teaching” (Teo, Chai, Hung, & Lee, 2008). Technology integration is the implementation of technology 
during teaching. Therefore, beliefs of preservice teachers about technology integration potentially influence 
their teaching methods when using technology. 

Teaching preservice teachers how to critically analyze their beliefs about technology use in classrooms 
influences their technology integration practices (Valcke, Sang, Rots & Hermans, 2010); that is, the 
pedagogical beliefs of preservice teacher directly predict their technology integration during practice teaching 
(Sang, Valcke, van Braak, & Tondeur, 2010. In addition, literature reveals that teacher education courses 
may shape pedagogical beliefs of preservice teachers and enhance their technology skills, rather than the 
ability to integrate technology while teaching. As a result, teacher education courses shape preservice teacher 
beliefs and, further, beliefs are predictive of technology integration and worthy of exploration. 


2. METHODS 
2.1 Participants 

The participants in this study included teacher candidates participating in a teacher preparation program 
designed to prepared special education teachers for a new license in the state of Minnesota. As part of the 
program, teacher candidates were enrolled in a newly designed course intended to integrate an introduction to 
special education, individual education programs, working with school personnel and learning about 
educational technology and how to integrate it into their teaching. There were twenty-five students in the 
cohort beginning January 2012; 22 participated in the pre-course survey, 24 in the post-course survey and 15 
in the survey at the end of the program. Eighty-seven percent of the participants were female. 
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2.2 Procedures 

SPED 401: Individual Education Program (IEP) Writing and Professional Practice is a blended course 
including face-to-face, computer lab sessions enhanced with online activities and discussions. The course 
was developed in alignment with Minnesota Board of Teaching (BOT) standards. Eighteen standards from 
three groups were included: Core Special Education, Academic and Behavioral Strategist and Standards of 
Effective Practice. The course is included in the first semester (Block 1) of the preparation program for 
students who want to become licensed in Minnesota’s new cross-categorical area. Academic and Behavioral 
Strategist (ABS). The ABS program prepares new special education teachers to work with students with 
mild/moderate learning disabilities, emotional/behavioral disorders, developmental disabilities and autism 
spectrum disorders. SPED 401 was developed using face-to-face, computer lab and online components and 
covers a variety of special education related topics while building in the knowledge, and use, of educational 
technology as a tool to enhance and support curriculum. This reinforces the promise the today’s teachers 
must have the 21 st Century skills needed to integrate technology into their teaching throughout their lessons. 
SPED 401 introduces teacher candidates to these concepts and practices, as well as, provides them with the 
foundational skills to do so. New tools were introduced and time was provided throughout the duration of the 
course for practice and support of how to use the tool in pedagogy. 

In an effort to address our research questions we provide teacher candidates with a pre-test to assess their 
perception of their own proficiency with hardware (e.g., Smart board, digital cameras, etc.) and software and 
uses of software (e.g., Movie Maker, PowerPoint, image capture software, etc.). In addition the survey 
addressed perceptions of the extent to which technology contributes to student learning and more. Our 
longitudinal study addresses the following four research questions: 1) What impact does participating in 
SPED 401 ( special education course that integrated educational technology ) have on teacher candidates’ 
perceptions of proficiency with hardware and software immediately following the course and two years 
later?; 2) How do teacher candidates report their knowledge and skills related to educational technology 
impact their teaching?; 3) What relationship exists between how teacher candidates perceive technology 
influences their teaching and their proficiency with educational technology tools?; and 4) To what extent do 
teacher candidates perceive k-12 student engagement is enhanced due to the integration of technology into 
teaching and learning? This paper will mostly focus on question one: What impact does participating in 
SPED 401 ( special education course that integrated educational technology ) have on teacher candidates’ 
perceptions of proficiency with hardware and software immediately following the course? Our initial time 
frame is two years after the SPED 401 course. 


3. RESULTS 

3.1 Description of Data Analysis 

The sample population (N) was small (N=15-22 teacher candidates) so the statistical tests used to examine 
differences had to be appropriate for the sample; therefore, a chi-square test was used to complete the 
statistics for the data set. The chi-square test is used to determine whether there is a significant difference 
between the expected frequencies and the observed frequencies in one or more categories using a chi-square 
value of less than .05. The data set used had numerical data comprised through Likert scale responses. Means 
for a subset of the items are shown in Table 1 (Hardware) and Table 2 (Software) for the pre-test (survey 
done before the course), post-test (survey done after the course) and follow-up (survey done at the end of the 
program at the completion of student teaching). The chi-square analysis compared counts of categorical 
responses between two independent groups. First a brief overview is provided for each research question and 
then the remainder of the paper and discussion focuses on research question 1 . 

In examining the initial findings, the data concluded that there was a significant change in students’ 
proficiency with camera usage between the pre-test and post-test, which addresses research question 1. The 
chi-square analysis showed positive significance in the extent to which students believe technology skills 
influence their learning when comparing results before the course and after their student teaching, which 
addresses research question 2. Further, a T-test was conducted evaluating questions within the follow-up 
survey questionnaire to address research questions 3 and 4. 
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This statistical measure was used to determine if there was a correlation between responses within the 
final survey given. The responses resulted in significance in student perspective that technology positively 
impacted teaching. Specifically, proficiency with Smart Board and moviemaker was positively impacted by 
the knowledge and skills they developed from the course. Finally, the perception that the skills they learned 
through SPED 401 enhanced student engagement from the integration of technology was statistically 
significant. The results found through the T-test show that the SPED 401 course had a positive influence in 
different areas of proficiency when integrating technology into the classroom. Further examination of the 
analyses will add meaning. 


Table 1 . Perceptions of proficiency with hardware - Pre-test, post-test and follow-up 


Hardware 

Pre-Test 

N=22 

Post-Test 

N=24 

Follow-up 

N=15 

Portable e-devices 

3.64 

3.79 

4.33 

Smartboard 

2.45 

4.00 

3.80 

Digital Camera 

4.27 

4.25 

4.33 

Cell Phone 

4.73 

4.79 

4.73 


Note. The ratings are based on a 5-point Likert scale with 5 the highest. 


Table 2. Perceptions of proficiency with software - Pre-test, post-test and follow-up 


Software 

Pre-Test 

N=22 

Post-Test 

N=24 

Follow-up 

N=15 

Online searches 

4.38 

4.33 

4.33 

Movie Maker 

2.24 

3.63 

3.14 

PowerPoint 

4.23 

4.63 

4.70 

Image Capture 

2.41 

3.79 

3.70 

Digital Information Transfer 

2.64 

3.71 

3.50 

Desire to Learn (D2L) 

4.55 

4.67 

4.86 

Facebook 

4.63 

4.71 

4.64 

Hypertext Linking 

2.14 

3.08 

3.21 

Twitter 

2.91 

3.17 

3.50 


Note. The ratings are based on a 5-point Likert scale with 5 the highest. 


In addition to the survey, formative and summative assessments were used to assess the technology and 
special education content learning of preservice teachers (teacher candidates) throughout the course. 
Formative assessments took place throughout the course using discussion groups, individual assignments, 
papers, and technology infused student products. In addition to assessments of student special education 
content learning where all students achieved 90% or higher, summative assessment came from comparative 
analysis of the pre and post student technology surveys. The technology survey contains four sections: 
Student demographics. Technology Training, Technology Skills & Usage, and Learning Preferences. It was 
clear teacher candidates were in favor of requiring teachers to take a technology related course. The three 
outcomes reported most often by the teacher candidates through the content analysis of the post-test survey 
suggested that teacher candidates (a) learned strategies to enhance the presentation of material to k-12 
students, b) learned more about ways to use technology in the classroom for teaching, and c) enhanced their 
skills at recognizing and using the technology tools available to them. Analysis of the narrative responses to 
the open ended questions on the follow-up survey has yet to occur. 


4. DISCUSSION 

While additional analysis is needed with the follow-up survey completed, it is clear that teacher candidates 
perceive the importance of learning about educational technology tools and the integration into their teaching 
as important and necessary. They reported that learning about and using educational technology integration 
had statistically significant positive impact on their teaching as well as enhancing the engagement of k-12 
students. Specifically, proficiency with Smart Board and moviemaker was positively impacted by the 
knowledge and skills they developed from the course. Hybrid, blended, and online course designs, along with 
motivational impetus for personalized learning mark other pedagogical approaches which need to be studied. 
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There are important implications for teacher preparation to include close collaboration between teacher 
education programs and field experience, focusing on specific technology uses. In support of the work of Lue 
(2011) and Kajder (2005) when teacher candidates were asked to reflect on the use of technology in the k-12 
classrooms they visited for their field experience, their individual reflections varied according to the mentor 
teachers they witnessed. For example, when asked how iPads were used to enhance student learning some 
teacher candidates indicated they were used to motivate students. Specifically, if a child completed an 
assignment, he or she was told he or she could use the iPad. However, in another classroom, that 
“motivation” was probably still true and the use of the iPad included a specific application intended to 
reinforce earlier instruction. These are both valuable but very different uses of technology for learning. It is 
clear teacher candidates must learn how to integrate the use of instructional technology in their teaching and 
it must begin in their preservice programs, and following the TPACK model of technology integration as the 
intersection of technological knowledge, content knowledge and pedagogical knowledge seems an 
appropriate and effective strategy. As suggested by Graham, Borup, & Smith (2011) it is the overlap of the 
three types of knowledge that allows for the efficacious integration of technology into pedagogy, which is the 
goal. 
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ABSTRACT 

Over the past years. Remote and Virtual Labs (RVLs) have gained increased attention for their potential to support 
technology-enhanced science education by enabling science teachers to improve their day-to-day science teaching. 
Therefore, many educational institutions and scientific organizations have invested efforts for providing online access to 
state-of-the-art science experiments via RVLs. Currently, there are existing initiatives for the storage and organization of 
existing RVLs into web-based repositories towards increasing their findability and enabling science teachers to search 
and retrieve them for further usage into their lesson plans. Nevertheless, most of these repositories adopt metadata models 
that store limited information related to the pedagogical context of their lesson plans. As a result, science teachers are not 
supported in selecting RVLs taking into consideration core elements of their lesson planning. In this paper, we aim to 
tackle this problem by proposing the ASK4Labs a web-based repository for supporting learning design driven RVLs 
recommendations. Preliminary evaluation results are also described, which indicate that the proposed recommender 
system can provide robust identification of appropriate RVLs based on the pedagogical context elements of the intended 
lesson plans. 


KEYWORDS 

Remote labs. Virtual Labs, Web-based Repositories, Metadata Model, Learning Design, Recommender Systems 


1. INTRODUCTION 

Over the past years, technological advancements in the field of World Wide Web have allowed the 
advancement of physical laboratories and their partial replacement by remotely-operated labs (“remote labs”) 
and virtual labs (de Jong et al. 2013; Balamuralithara & Woods 2009). More specifically, remote labs provide 
students with the opportunity to collect data from a real physical laboratory, including real equipment from 
remote locations (Gomes & Bogosyan 2009). On the other hand, virtual labs represent interactive 
environments for designing and conducting simulated experiments (Balamuralithara & Woods 2009). Both, 
Remote and Virtual labs (RVLs) have gained increased attention for their potential to support technology- 
enhanced science education by enabling science teachers to improve their day-to-day science teaching. 
Additionally, it has been shown that RVLs are more effective in increasing sUidents’ interest in science and 
their engagement in related learning activities compared to Uaditional laboratories (Jaakkola et al. 2011; de 
Jong 2010; Kong et al. 2009). Thus, many educational institutions and scientific organizations have invested 
efforts for providing online access to state-of-the-art science experiments via RVLs (Gravier et al. 2008). 

Nevertheless, most RVLs currently available online are still scattered around the web. As a result, in 
order to increase their findability and enable science teachers to search and reUieve them for further usage, 
there are existing initiatives for their storage and organization into web-based repositories (Richter et al. 
2011; Maier & Niederstatter 2010). However, existing RVL repositories are adopting different metadata 
models for characterizing their RVLs. Furthermore, most of these repositories adopt metadata models that 
store limited information related to the pedagogical context of their lesson plans such as the teaching 
approach adopted, the subject domain, the intended educational objectives and the grade level. As a result, 
science teachers are not supported in selecting RVLs taking into consideration core elements of their lesson 


172 



1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


planning. Within this context, the aim of this paper is to tackle this problem by proposing the ASK4Labs a 
web-based repository for supporting learning design driven RVLs recommendations. Moreover, preliminary 
evaluation results are described, which indicate that the proposed recommender system can provide robust 
identification of appropriate RVLs based on the pedagogical context elements of the intended lesson plans.. 

The paper is structured as follows. Following this introduction, section 2 reviews existing RVLs 
repositories and their metadata models and identifies their limitations related to the pedagogical information 
that they store. Section 3 presents the proposed ASK4Labs repository by presenting its main functionalities 
and the recommender system that it incorporates. Section 4 comprises the evaluation of the proposed 
recommender system. Finally, we discuss our main conclusions and ideas for further work. 


2. REVIEW OF EXISTING REPOSITORIES OF REMOTE AND 
VIRTUAL LABS 


According to Conole & Fill (2005) there are two (2) dimensions that constitute a learning design: 

■ The pedagogical context within which the learning design occurs. This includes, among others, the 
following elements: (a) the subject domain (i.e., physics, geography, math, arts, etc.), (b) the 
intended educational objectives (i.e., recall, understand, etc.), (c) the grade level (i.e. primary 
education, secondary education, higher education etc.) and (d) the teaching approach adopted (i.e., 
problem based learning, inquiry based learning, etc.). 

■ The learning activities undertaken to achieve the intended educational objectives. 

Thus, it is important that the pedagogical context elements are accommodated by the metadata model 
adopted by a RVL repository. This is essential in order to facilitate search and retrieval of RVLs taking into 
consideration elements of the pedagogical context of learning designs. 

In previous work, a review of existing repositories of remote and virtual labs was performed in order to 
highlight the metadata models adopted by existing RVLs repositories. Furthermore, we conducted a 
comparative analysis of the elements used by the metadata models of these repositories (Zervas et al. 2014). 
In the context of the present study, a meta-analysis of those results was performed in order to identify for 
each metadata model those metadata fields that store information related to the pedagogical context elements 
of a learning design. Table 1 presents the results of our findings. 

Table 1 . Overview of Existing RVLs Repositories and their Metadata Models 




Type of Labs 

No 

RVLs Repositories 

Remote 

Labs 

Virtual 

Labs 

1 

PhET 1 

- 

V 

2 

3 

Library of Labs 2 
Labshare 3 

V 

V 

4 

5 

Open Sources 
Physics 4 
Smart Science 5 

V 

V 

6 

Molecular 

Workbench 6 

- 

v 

7 

8 

Explore Learning 7 
ChemCollective s 

- 

V 

V 

9 

Remotely Controlled 
Laboratories (RCL) 9 


- 


Learning Design Pedagogical Context Elements 


Subject 

Domain 

Educational 

Objectives 

Grade 

Level 

Teaching 

Approach 

V 

V 

V 

- 

V 

- 

- 

- 

V 

- 

- 

- 

V 

- 

V 

- 

V 

V 

- 

- 

v 

- 

- 

- 

V 

V 

V 

_ 

V 

- 

- 

- 

_ 

_ 

_ 

_ 


1 http://phet.colorado.edu 

2 https://www.library-of-labs.org/ 
1 http://www.labshare.edu.au/ 

4 http://www.compadre.org/osp 

5 http://www.smartscience.net/ 

6 http://mw.concord.org/ 

7 http://www.explorelearning.com 
s http://www.chemcollective.org/ 
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10 

Skoool 10 

- 

V 

V 

- 

V 

- 

11 

iLabCentral 11 

V 

- 

V 

- 

V 

- 

12 

Lab2Go 12 

V 

V 

V 

- 

- 

- 

13 

WebLab Deusto 13 

V 

- 

V 

- 

- 

- 


As we can notice from Table 1, 12 (92,30%) of the examined repositories characterize their RVLs based 
on the subject domain, whereas only 5 (38,46%) characterize their RVLs based on the grade level. Moreover, 
only 3 (23,07) of the examined repositories adopt a metadata model that stores information about the 
educational objectives that their RVLs address. Finally, none of the examined repositories includes 
information about the teaching approach that their RVLs can be used. 

Following this analysis, we can identify that none of the examined repositories support all learning design 
pedagogical context elements. We consider this a major shortcoming in facilitating search and retrieval of 
RVLs based on their pedagogical characteristics. Within this context, our research problem is the design and 
evaluation of a recommender system that aims to support science teachers in selecting RVLs stored in a web 
repository taking into consideration pedagogical characteristics for given lesson plans. Our proposed solution 
is presented in this paper with the ASK4Labs web-based repository of RVLs that incorporates (a) an 
appropriately designed metadata model and (b) a recommender system that addresses this problem. 


3. THE ASK4LABS WEB-BASED REPOSITORY 

The ASK4Labs is a web-based repository that provides access to RVLs. It has been developed based on 
Drupal, which is a widely used, open source content management system and content management 
framework based on PHP and MySQL. For the purpose of our research, the ASK4Labs repository was 
populated with 45 RVLs. In the next paragraphs, we describe (a) the metadata model of the ASK4Labs 
repository, (b) its main functionalities and (c) the recommender system that incorporates the metadata model 
for facilitating selection and retrieval of appropriate RVLs based on the pedagogical context elements of 
given learning designs. 

3.1 Metadata Model 

The ASK4Labs repository adopts a metadata model that has been described in previous work (Zervas et al. 
2014). More specifically, the starting point for developing this metadata model was the outcomes of an 
extensive review of the metadata models of existing repositories of RVLs. Additionally, we considered for 
our model metadata elements that store information about the pedagogical context of a learning design, as 
described in section 2. Table 2 presents the metadata elements of the proposed metadata model. 


Table 2. ASK4Labs Repository Metadata Elements (Zervas et al. 2014) 


No 

Metadata 

Group 

Metadata 

Sub-Group 

Element Name 

Description 

Taxonomy 

Available? 

1 



Title 

Refers to the complete title of the lab 

No 

2 

3 

4 

General 

Metadata 

- 

Description 
Lab Type 

Language 

Provides a textual description of the lab 
Refers to the specific kind of the lab 
Refers to the languages that the lab is 
available in 

No 

YES 

YES 

5 



Keyword 

Refers to a set of terms that characterize the 
content of the lab 

No 

6 

7 

Organizational 

Metadata 

- 

Access Rights 
Rights Holder 

Refers to the lab’s access permissions 
Refers to those entities that hold the lab’s 

YES 

No 


9 http://rcl-munich.informatik.unibw-muenchen.de 

10 http://skoool.com 

1 1 http://ilabcentral.org 

12 http://www.lab2go.net 

13 https://www.weblab.deusto.es/weblab/client/#page=home 
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Lifecycle Date 

copyrights 

Refers to critical dates related to the lab’s 

No 

8 



lifecycle 

Provides information about contact details 



9 



Contact Details 

of the person or the organization responsible 
for the lab 

No 

10 



Cost 

Refers to any payment required for using the 
lab 

YES 

11 

12 



Licence 

Provider 

Provides information about copyrights and 
restrictions applied to the use of the lab. 
Provides information about the provider of 
the lab. 

Refers to each person (or entity) that has 

YES 

No 

13 



Contributor 

contributed in the making of the lab in its 
current state 

No 

14 



Version 

Provides information about the current 
version of the lab 

No 

15 



Status 

Provides information about the availability 
status of the lab 

YES 

16 


Learning 

Teaching Approach 

Refers to the teaching approach where the 
lab can be used 

No 

17 


Design 

Subject Domain 

Refers to the lab’s subject domain 

YES 

18 


Pedagogical 

Grade Level 

Refers to the grade level for which the lab 

YES 

Pedagogical 

Context 

can be used 

19 

Metadata 

Metadata 

Educational 

Refers to the educational objectives that the 

No 



Objectives 

lab addresses 

20 



Level of Difficulty 

Refers to the level of difficulty of the lab 

YES 

21 


- 

Intended End User 

Refers to the principal users for whom the 

YES 



Role 

lab was designed 


22 



Location URL 

Provides a URL for accessing the lab 

No 

23 

Technical 


Technical 

Refers to the technical requirements that are 

YES 

Metadata 


Requirements 

needed for using the lab. 

24 



Technical Format 

Refers to lab’s technical format. 

YES 

25 



Student’s Resource 

Refers to the type and the URL of student’s 
resource that is connected to the lab 

No 

26 

Content 

Metadata 

- 

Teacher’s Resource 

Provides the URL for accessing any lesson 
plan that can be used for exploiting the lab. 

No 

27 



Supportive App 

Provides the URL for accessing any 
supportive app that is connected to the lab. 

No 


3.2 Main Functionalities 

The ASK4Labs Repository targets the following user groups: 

■ RVLs owners, who want to characterize their RVLs with metadata and store them to the ASK4Labs 
repository, so as to increase their visibility and share them with science teachers for further usage 
into their day-to-day teaching activities. 

■ Science teachers, who want to search and find RVLs for using them into their lesson plans 
The main functionalities of ASK4Labs Repository can be summarized as follows: 

Store RVLs: RVLs owners are able to store in the ASK4Labs repository their RVLs along with metadata 
descriptions following the metadata model described in section 3.1. Figure 1 presents the process of storing a 
RVL to the repository by completing the appropriate metadata fields . 
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Complete 

General 

Metadata 




▼ Pedagogical 
Subject * 


Educational Level * 

Primary Education 

Upper Secondary Education 
Higher Education 


Difficulty * 

- Select a value - * 



Complete 

Pedagogical 

Metadata 


Show row weights 

Learning Objectives 

I *** 

Add another item 


Figure 1. Store a RVL to the ASK4Labs Repository by characterizing it with metadata 


Search for RVLs: Science teachers are able to search, browse and retrieve RVLs by using terms, which 
are matched with metadata descriptions of RVLs. Moreover, the searching mechanism incorporates a 
recommender system (it will be further described in section 3.3), which enables science teacher to receive 
recommendations about RVLs based on the pedagogical context elements of their lesson plans. Figure 2 
presents the process of searching RVLs in the ASK4Labs repository. 

View RVLs’ Metadata: Science teachers have the capability to view in details the metadata descriptions 
of RVLs, so as to be able to decide whether to use or not a specific RVL. Figure 3 presents the educational 
metadata of a selected RVL. 


ASK 4 Labs 

i epos limy of i 


& femote lobs 


Filter search 
results 


About S/jrch labs Contact 


Search results 


Lab Type 

• Virtual lab (33) 


Subject 

o Biology (5) 

► Astronomy (4) 

• Math (3) 

• Density (2) 

• Betaradtaoon(l) 

• 

• Electromagnetism (H 

• Boating and sinking (1) 
< Gamma radiation (1 1 


Online Labs List 


jravlty Force Lab 


Lab Type: Virtual Lab 
Subject: Physics. Motion 

Educational Level: Pnmaiy Education. Lower Secondary 
Education 

LanguagMtL English, Greek. Spanish. French. German 


Difficulty: Medium 
Keywords: Force. Gravity 

Cost: no 

Booking Required: no 

Description: Visualize the gravitational force that two objects exert on each other. Change 
properties of the objects in order to see how it changes the gravity force. 



Figure 2. Search RVLs Figure 3. View RVL Metadata and Provide Ratings / Comments 

Rate/Comment RVLs: Science teachers are able to provide their ratings and comments for the RVLs 
stored in ASK4Labs Repository. These ratings and comments can be related to the impressions of the science 
teachers who have used a specific RVL. Figure 3 presents the process of providing ratings/comments to a 
selected RVL. 

3.3 Recommender System 

The recommender system that has been incorporated in the ASK4Labs repository is a content-based 
recommender system that uses Vector Space Model (VSM) with basic TF-IDF (Term Frequency-Inverse 
Document Frequency) weighting (Lops et al. 2011). The recommender system aims to provide an ordered 
list of RVLs based on the learning design pedagogical context elements that are used during search by the 
science teachers. 
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More specifically, the pedagogical context of a learning design can be modelled as follows: 
Learning_Design l = { LD ,, LD 2 , LD 3 , LD 4 } where LD, represents the elements of the pedagogical context of a 
learning design. Additionally, for each element a different weighting factor can be applied namely Weight , = 
{ Wj, w 2 , tv ( , w 4 ) corresponding to each learning design vector space model elements. Different instantiations 
of the weight vector space model can be defined manually by each science teacher. 

On the other hand, a RVL can be modelled based on the different metadata elements already described in 
section 3.1, as follows: Metadata y = { MD h MD 2 , ...., MD 27 ), where MD, corresponds to each of the RVL 
metadata elements. However, not all metadata elements can be exploited in the context of the specific 
recommender system because not all of them can be mapped to the learning design vector space model 
elements. Thus, we performed a mapping between the learning design vector space model elements and the 
RVL metadata vector space model elements. Additionally, for the purpose of our research, we assigned 
percentages of relevancy for each metadata element that was mapped to the learning design pedagogical 
context elements. Table 3 presents this mapping and the assigned percentages of relevancy. 


Table 3. Learning Design Pedagogical Context Elements mapped to RVL Metadata 


Learning Design 
Pedagogical Context 
Elements 

Percentage of Relevancy 

RVL Metadata 

Subject Domain 

60% 

20% 

10% 

10% 

Subject Domain 

Keyword 

Title 

Description 


70% 

Learning Objectives 

Educational Objectives 

20% 

10% 

Keyword 

Description 

Grade Level 

80% 

20% 

Grade Level 
Description 

Teaching Approach 

70% 

30% 

Teaching Approach 
Description 


Based on the mapping presented in Table 3, the RVL metadata vector space model can be limited to 7 
distinctive elements as follows: Metadata y = { MD h MD 2 , MD 7 } where MDi=title, MD 2 =description, 
MD 3 =keyword, MD 4 =subject domain, MD 5 =Educational Objectives, MD 6 =grade level and MD 7 =teaching 
approach. 

According to the above modelling, next we present the process of generating the RVL recommendations 
in pseudo-code. As we can notice, the recommender system calculates an overall score for each RVL based 
on the query terms provided by the science teacher. It should be noted that we consider four query terms, 
each related to a separate Learning Design Vector Space Model element. Based on the calculated score, the 
RVLs are ranked from the most relevant to least relevant. 


For each query_term in LDi field { 

If (LDi = teaching approach) then { 


If 

query term found in MD 7 
Score+= Wj* tf-idf 

(query term. 

MD„, #RVLs) 

* 0,7 

If 

1 

query term found in MD 2 
Score+= Wj* tf-idf 

(query term. 

MD 3 , #RVLs) 

* 0,3 

If (LDi = grade level) then { 

If query term found in MD 6 
Score+= Wj* tf-idf 

(query term. 

MD 4 , #RVLs) 

* 0.8 

If 

query term found in MD 2 
Score+= Wj* tf-idf 

(query term. 

MD 3j #RVLs) 

* 0.2 
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4. PRELIMINARY EVALUATION 

4.1 Method 

An experiment was performed towards the initial evaluation of the proposed recommender system. More 
specifically, the focus of the experiment was to evaluate the ranking accuracy of the recommender system 
utilizing the spearman’s rank correlation coefficient (Shani & Gunawardana, 2011). The evaluation 
methodology that was conducted included the following steps: 

Step 1: two queries were created representing two different pedagogical contexts for a learning design, as 
follows: Learning_Design 1 = {subject domain = "chemistry", educational objectives = "learn to carry out 
tests with chemical solutions", grade level = "lower secondary education", teaching approach = "inquiry 
based learning"), Learning_Design 2 = {subject domain = "physics", educational objectives = "learn about 
forces and balance", grade level = "primary education", teaching approach = "problem based learning"). 
Additionally, three instances of the weight vector space model were created, namely Weight j = {0.5, 0.2, 0.2, 
0.1} (emphasis given to the subject domain). Weight 2 = {0.2, 0.2, 0.5, 0.1) (emphasis given to the grade 
level). Weights = {0.2, 0.5, 0.1, 0.2} (emphasis given to the educational objectives). For both queries, each 
instance of the weight vector space model was applied and run at the ASK4Labs repository. Finally, the 
recommender system provided us with six ranked lists of RVLs. 

Step 2: we asked 30 secondary education science school teachers to validate the ranked lists produced 
from the recommender system. Each of the teachers was given the option to agree with the ranking or to 
propose his/her own ranked list. 

Step 3: Finally, we calculated the spearman’s rank correlation coefficient between the ranked lists 
produced by the recommender system and the ones provided by the science teachers. 

4.2 Results 

The preliminary evaluation results for ranking accuracy of the recommender system are presented in Figure 



Figure 4. Spearman’s Rank Correlation Coefficient per Ranked List (*= p<0,05, **=p<0,01) 

As we can notice from Figure 4, the spearman’s rank correlation coefficient was very high for all ranked 
lists produced by the recommender system. This provided us with evidence about the validity of our 
proposed mapping between leaning design elements and the RVL metadata elements, as well as about the 
usefulness of the percentages of relevancy for this mapping, as presented in Table 3. 

Furthermore, it is worth to notice that the ranking accuracy of the recommender system achieved the 
highest values for queries, where the emphasis to the weighting factors was given to those learning design 
pedagogical context elements that are mapped to RVL metadata modelled with taxonomies (see Table 2) 
compared to those modelled as free text elements. This can be explained by the fact that when metadata 
values are added as free text, polysemy and synonymy could be increased and the error margin of the 
recommender system could be also increased. 
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5. CONCLUSIONS AND FUTURE WORK 

In this paper, it was argued that there is a growing trend for the development of web-based repositories that 
provide access to RVLs. However, existing RVLs repositories are adopting different metadata models for 
characterizing their RVLs and these metadata models store limited information related to the core elements 
of lesson planning. Therefore, we presented the ASK4Labs, a web-based repository for supporting learning 
design driven RVLs recommendations. A preliminary evaluation was also performed, which focused on the 
ranking accuracy of the proposed recommender system. The results showed a high level of ranking accuracy 
especially when the weighting factors were assigned with emphasis given to learning design pedagogical 
context elements mapped to RVL metadata elements modeled with taxonomies. 

Future work includes additional evaluations of the ranking accuracy of the proposed recommender system 
with more users and with more search queries including different combinations of the weighting factors. 
Moreover, the recommender system could be enhanced by taking into account ratings of the users towards 
providing more accurate rankings of RVLs. 
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ABSTRACT 

In tandem with the deep structural changes that have taken place in society, education must also shift towards a teaching 
approach focused on learning and the overall development of the student. The integration of technology may be the drive 
to foster the needed changes. We draw on the literature of pertaining to the role of emotions and interpersonal 
relationships in the learning process; the technological evolution of storytelling towards Digital Storytelling and its 
connections to education. We argue Digital Storytelling is capable of challenging HE contexts, namely the emotional 
realm, where the private vs. public dichotomy is more prominent. Ultimately we propose Digital Storytelling as the 
aggregator capable of personalizing Higher Education while developing essential skills and competences. 
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1. INTRODUCTION 

In the complex society we live in, with the unforeseen future demands and the need for competence 
development, it has become widely acknowledged that approaches to teaching and learning need to 
encourage greater student involvement anchored in constructivist perspectives. As Laurillard (1993), among 
others, has argued, higher levels of thinking and cognitive development occur in contexts that stimulate 
curiosity, problem solving and reflective, critical thinking skills (see also the work of Schon, Kolb, and 
Moon), where students are actively engaged in learning, in the construction of knowledge (see the work of 
Dewey, Freire, and Vygosky, for example). In the foreword of the book Education for judgment: the artistry 
of discussion leadership, Elmore (1991) states: 

The aim of teaching is not only to transmit information, but also to transform students from passive 
recipients of other people's knowledge into active constructors of their own and others' knowledge. The 
teacher cannot transform without the student's active participation, of course. Teaching is fundamentally 
about creating the pedagogical, social, and ethical conditions under which students agree to take charge of 
their own learning, individually and collectively (p. xvi-xvii) 

For many teachers in higher educational contexts, the challenge lies in attempting to understand the 
emerging educational context and the creation of learning environments that will make the development of 
higher-order cognitive abilities possible while encouraging teachers and students to thrive in what has been 
said to be the new technological paradigm: informationalism (Castells 2000). The integration of technology 
in education has been acknowledged to bring forth positive student engagement on all educational levels 
(Bates and Bates 2005, Latchman et al. 1999, Laurillard 1993). As students become not only consumers but 
also active content creators, and literature demonstrates that technological integration in HE may constitute 
an interesting strategy to motivate student learning (see Laurillard 1993, Rogers 2000, Bates and Poole 2003, 
Daniel 1998, Garrison and Kanuka 2004), it invites the question whether digital technology, particularly 
Digital Storytelling (DS) can possibly foster a more personalized Higher Education (HE). However, getting 
personal in HE, especially through stories seems to give raise to conflicting views. Based on the literature, 
we analyze and discuss emotion, interpersonal relationships and storytelling in order to seek further 
understanding regarding the possible reasons for this contradiction and argue Digital Storytelling might be a 
feasible approach to reemerge the emotional and personal in HE. 
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2. GETTING PERSONAL IN HIGHER EDUCATION 

Thirty years of research have allowed Pascarella and Terenzini (1991, 2005) to conclude that “Modern 
colleges and especially universities seem far better structured to process large numbers of students efficiently 
than to maximize student learning" (p. 646), given that there are other essential dimensions beyond the 
cognitive skills and intellectual growth that HEIs that are still lacking. These include consideration of 
students’ psychosocial changes, related to identity and self-concept; those related to others and the world; 
those related to values and attitudes; and those related to moral development. If HE is to be viewed as a 
facilitator for positive overall student development, all stakeholders involved need to rethink learning to 
include more than scientific knowledge. Illeris (2003) conceptualized this interplay of multiple dimensions 
and processes into a model of leaning. Illeris (2003, 2008) claims learning implies a series of processes that 
"lead to relatively lasting changes of capacity, whether they be of a motor, cognitive, psychodynamic (i.e. 
emotional, motivational or attitudinal) or social character, and which are not due to genetic -biological 
maturation” (2003, p. 397). Illeris’ definition of learning demonstrates that it cannot be separated from 
personal development, socialization and qualification. The author explains that learning implies the 
integration of two processes - an external interaction process between the learner and his or her social, 
cultural or material environment, and an internal psychological process of acquisition and elaboration - and 
three dimensions - the content dimension, usually described as knowledge and skills, but also many other 
things such as opinions, insight, meaning, attitudes, values, ways of behavior, methods, strategies, and so on; 
the incentive dimension which comprises elements such as feelings, emotions, motivation and volition and 
whose function is to secure the continuous mental balance of the student; and the interaction dimension, 
which serves the personal integration in communities and society and thereby also builds up the student’s 
social dimension. 

Illeris draws on the work developed by Vygotsky (1978) and Furth (1987), who acknowledged the 
connection cognition and the emotion, and that of Damasio (1994, 2000) who has more recently proven that 
both cognition and emotion are always involved in the learning process. While cognition is connected to 
meaning making, the emotional content, Illeris defends, secures mental balance. The social dimension’s main 
function is personal integration in communities and society. Other scholars who recognize this three 
dimensional interplay in learning - meaning, personal (self and identity) and contextual interaction - are 
Lave and Wenger (1991), in what they describe as situated learning and Wenger (1998), on communities of 
practice, where learning is perceived as “a way of being in the social world, not a way of coming to know 
about it” (Hanks, 1991, p.??). While cognition is embraced and nurtured in HE, emotion and close 
interpersonal relationships are aspects that, despite the literature advocating their relevance, still tend to be 
disregarded in favor of more traditional approaches to teaching and learning, as these are considered private 
and beyond the scope of HE (Leathwood and Hey, 2009, Morley, 2003, Clark, 1983). Thus, regardless of the 
current emphasis on student-centered learning approaches, considerable effort is made to maintain the firmly 
established boundaries and the distance deemed necessary. 

2.1 Situating Emotion 

Stones (1978) was amongst the first scholars to talk about the convergence of psychology and teaching, in 
what he termed as psychopedagogy (p. 1), which means applying theoretical principles of psychology into 
teaching, in order to enhance teaching and its affective context, establishing a link between cognition and 
emotion. Although current literature often tends to associate psychopedagogy with learning problems, 
Saravali (2005), for example, recognizes the role of psychopedagoy in HE, where teachers are asked to 
facilitate meaningful learning at a time when students of all ages face personal development challenges, as 
we have seen. Saravali admits knowledge on student development and pedagogy is useful to help students, 
both socially and affectively. Emotions are essential for human survival and adaptation as they affect the way 
we see, interpret, interact and react to the world that surrounds us (Horsdal 2012). Boler (1999) admits 
emotions are underexplored in education. We concur with the author that it is not that pedagogy of emotions 
should prevail, and that teachers and students should disclose their innermost secrets and feelings to each 
other in the classroom. As teachers we do need to be aware of the intrinsic implicit and explicit relations in 
higher educational settings and consider the reasons why emotions have systematically been discouraged at 
this educational level. 
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Boler (1999) claims emotions are embodied and situated, in part sensational and physiological, consisting 
of actual feeling - increased heartbeat, adrenaline - as well as cognitive and conceptual, shaped by beliefs 
and perceptions. The author identifies three deeply embedded conceptions surrounding emotions, which may 
allow us to better grasp the reasons behind the apparent duel. Emotions have been conceived as private 
experiences people are taught not to express publicly; they are a natural phenomenon people must learn to 
control; and are an individual (intimate) experience. Finally, emotion has been excluded from the HE’s 
pursuit of truth, reason and knowledge. To address emotion is risky business, especially when, as the author 
argues, reason and truth prevails in HE. Emotions still tend to be associated with what the author describes as 
“‘soft’ scholarship, pollution of truth and bias” (Boler, 1999, p.??), despite the proliferation of recent findings 
from the neurosciences advocating emotions as natural and universal. 

In his theory of consciousness, neurobiologist Damasio (2000) argues feelings and high-level cognition 
are intimately connected. The author claims a person’s emotions can either inhibit or foment the brain’s 
rational functioning. Additionally, consciousness of the world and of the self emerge in the same process. 
Damasio (2000) explains: “the presence of you is the feeling of what happens when your being is modified 
by the act of apprehending something” (p. 10). Thus, all that occurs to a person is emotionally laden. 
Damasio links not only cognition and emotion, but also the process of meaning making, or learning. Given 
the significance of this finding, the last ten years has seen an increase in the literature on emotions in 
education. Schutz and Lanehart (2002) state “emotions are intimately involved in virtually every aspect of the 
teaching and learning process and, therefore, an understanding of the nature of emotions within the school 
context is essential” (p. 67). Immordino-Yang and Damasio (2007) emphasize the bound relationship 
between emotion, learning and context in their recent article We feel, therefore we learn, where they discuss 
the relevance of emotions and social context on learning. The authors claim: 

Modem biology reveals humans to be fundamentally emotional and social creatures. And yet those of us 
in the field of education often fail to consider that the high level cognitive skills taught in schools, 
including reasoning, decision making and processes related to language, reading, and mathematics, do not 
function as rational, disembodied systems, somehow influenced but detached from emotion and the body. 

(p. 3) 

As empirical studies proliferate and claim positive connections between emotion and learning in HE, 
some authors recommend a cautious approach and alert to the risks involved. Rai (2012) examined the 
significance of emotion in assessment through reflective or experiential writing in the context of professional 
practice-based learning. The author found that reflective writing raises important issues in relation to emotion 
for both students and teachers assessing their texts. While admitting the advantages of personal, emotionally 
laden reflective writing, Rai adverts to the full complexity of the impact of emotions. Tobin (2004) also 
explores some of the academic literature focusing on writing personal reflective accounts and contends that 
while teachers should encourage emotion in the classroom, there is a degree of risk. For Tobin (2004) and 
Rai (2012) personal, reflective writing translates into a focus on emotions, in line with Schon’s (1983) view 
of reflection as an emotional process. On this account, Brantmeier (2013) also claims learning that involves 
reflective critical-thinking activities allows students to be flexible and fluid, responsive to future yet 
unforeseen contextual needs. The author admits emotions invite vulnerability that, despite the risks discussed 
previously, is able to deepen learning. Brantmeier argues the dialogic learning process should be based on the 
following premise; share, co-learn, and admit you do not know. Closer personal relations, whether between 
students or between students and teachers, step beyond the confines of what has traditionally been deemed as 
appropriate for HE. Personal or emotional aspects are met with mental barriers that pose difficulties to 
overcome but necessary to manage. 


3. THE INTERCONNECTED THREAD OF DIGITAL STORYTELLING 

Traditional storytelling and educational technology can be said to have travelled divergent paths in education. 
While technology has seeped relentlessly into classrooms of all grade levels, storytelling seems to be 
imprisoned in lower grade levels (K-4), and the remaining grade levels continue to intently pursue 
Portuguese and Mathematics with a strict focus on standardized, national assessment. This system pervades 
HE. However, research has, time and again, demonstrated the connection between storytelling and higher- 
order thinking skills (Bruner, 1990, 2004; McAdams, 1993, 2001, 2008). Stories are essential to human 
communication, learning and thinking. Sarbin (1986) proposed the “narratory principle: that human beings 
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think, perceive, imagine, and make moral choices according to narrative structures” (p. 8). This is 
corroborated by neuroscience and nemo -imaging studies, which validate the claims that stories activate brain 
activity associated with cognitive processes (see for example, Fletcher et al., 1995, Gallagher et al., 2000, 
Mar, 2004). It is through stories that experiences gain meaning (Bruner, 1990, Polkinghorne, 1988) and, 
through reflection and interpretation, is then transformed into knowledge (Schon, 1983, Lave and Wenger, 
1991). Stories enable the audience to learn by analogy, instead of direct experience (Jonassen and 
Hernandez-Serrano, 2002, Witherell and Noddings, 1991). Through storytelling, memory structures are 
construed (Schank, 1990, 1995) becoming easier to recall than scattered pieces of information. Schank 
describes intelligence as the “telling of the right story at the right time in the right way” (1990). Storytelling 
derives from the recollection and interpretation of an experience that has been significant; otherwise it is not 
remembered (Bruner, 1990, Schank, 1995, Thorndyke 1977, 1990). It is this dialogic activity in storytelling 
process that enables learning and thus, human development. Learning occurs when reflection on experience 
is then transformed into a logical, meaningful story that is shared with others (Clark, 2010; Clark & Rossiter, 
2008). This frames leaning as a social, experiential, reflective process, integrating the cognitive, emotional 
and social dimensions that Illeris (2003, 2008) identifies as essential to learning. From the author’s 
perspective, stories, especially personal stories, motivate and engage the author in the act of creation. To 
create a coherent and effective story, the author must carefully reflect, select, prioritize and organize what 
he/she wants to say and how this can be conveyed. As the story is told, the audience interprets, reflects and 
connects to their own personal experience, construing new (mental) stories or reinterpreting older stories, in 
order to construe new ones. Furthermore, if interaction is possible between author and audience, or amongst 
the audience this (social) interaction fosters discussion and further reflection. The entire process is mediated 
by the intervenients ’ prior knowledge, their feelings in addition to the social and cultural context. 

Despite the perceived value in storytelling, Cooney et al. (1998) have argued that once students reach 
functional literacy, story is cast aside, and regarded as an informal and recreational practice, not longer an 
essential skill for students. Pagnucci (2004) also posits while scholars promote the value of story writing, the 
academy often devalues narrative. This idea expressed by Bendt and Bowe (2000) summarizes what we 
believe is commonly accepted amongst educators, “Storytelling can ignite the imagination of children, giving 
them a taste for where books can take them. The excitement of storytelling can make reading and learning 
fun and can instill a sense of wonder about life and learning” (our emphasis, p. 1). 

The authors identify the advantages of storytelling, but associating it to a particular timeframe, when 
entertainment in education is socially acceptable. This has repercussions on higher levels of education. 
Stories, especially personal stories, tend to be subjective and emotional. In fact, what is most significant in 
storytelling is the premise that most significant learning takes place during or after powerful emotional events 
(Witherell and Noddings, 1991). Whereas some regard the emotion in storytelling as powerful, others deem 
emotion as a weakness. Crafting a personal story is a highly complex and engaging activity for meaning 
making that couples cognition and affection, and links the self to others. Stories are used to create 
consistency, clarification and coherence of the self, through subjective interpretation. Some criticize 
emotional and personal content in HE. However, research has repeatedly demonstrated the emotional content 
at the core of personal storytelling is connected to intelligence and higher cognition. It is a highly reflexive 
and recursive process which incorporates the essence of human development, identity and education. By 
adding the digital to personal storytelling, we are able to incorporate the technical aspects, which drive the 
information society we live in. 

Digital Storytelling allows conjugating storytelling and the latest technologies accessible to our students 
for learning purposes. DS addresses story in its multiple, interrelated elements, as well as visual and media 
literacies. Literature review reveals that DS and the inherent construction process engages and motivates 
students (Fletcher and Cambre, 2009, Lowenthal and Dunlap, 2010, Mcdrury and Alterio, 2003, Robin, 2008, 
Sadik, 2008, Sandars et al., 2008). 

We posit DS is the adhesive force capable of aggregating what research has identified as core. DS is 
capable of integrating different literacies and language skills, as it combines multimedia researching, 
production and presentation skills with more traditional activities like writing and oral production skills. In 
practice, DS compels students to interpret, organize, prioritize, and make meaning of scattered events. 
Students are forced to reflect on their relationship with themselves and their relation to others. The 
preparation and creation phase requires students to search for and collect audio and visual materials, such as 
images, photos and soundtracks, to support their story and then combine and organize them in such a way 
that allows them to create the effect they want. It obliges students to think critically about the meaning and 
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effectiveness of multiple modes (elements) and their combination. This also confronts students with 
copyright issues on the Web. The narrative function allows students to tell a story with their own voice. 
Students need to reflect and decide on what to disclose. They are able to record and edit their stories as often 
as they want before finally presenting them to their teachers and colleagues, thus being able to improve their 
work until it is to their liking. DS is a personal self -representation, mediated by its limits. Length restrictions 
foster new ways of thinking, creativity and imagination. DS is also user-generated media, placing the focus 
on the student instead of the teacher, giving students leeway to cater to their own individual interests and 
learning styles, toward a more personalized learning context. This however, changes classroom dynamics and 
relationships, putting a spin in traditional lectured-based HE classrooms. 

During the final viewing students may be confronted with positive or negative feedback to their final 
stories (as for example happens with movies uploaded onto YouTube). As a result, the sharing process is, as 
Malita and Martin state, “an excellent way to foster self-expression and tolerance, and to create an engaged 
community of learners”, as students are “actively engaged in the exchange of ideas, the asking and receiving 
for feedback, the learning in an informal and, concomitantly, in a familiar way about their topics of interest, 
from peers, (older) colleagues” (2010, p.??). This fosters further reflection, interpretation and meaning 
making in the author and the audience. The story circle and the story show are about listening, promoting 
community, trust and closer emotional ties between teacher and student and amongst the students. The 
content is personal and emotional, and thus empowering, motivating and engaging. It seems that Digital 
Storytelling offers more than an opportunity to incorporate technology. As a process. Digital Storytelling 
demonstrates the capacity to aggregate the essence of HE: human (personal) development, social relational 
development, and technology. 


4. CONCLUSION 

DS is not just about creating digital stories; the foundations are embedded in story telling , in the act of 
sharing. DS in education can foster closer interpersonal connections based on trust, affection and dialogue. 
The act of sharing begins in the Story Circle and continues through the Story Show. Significant cognitive 
development takes place in the interpersonal interactions prior to and after the act of creating the final story 
where self-reflection is the steppingstone to dialogue, as advocated by the literature. This process fosters 
opportunities to connect and deepen relationships between students and teachers and amongst students. On 
the other hand, for students to talk about what is socially perceived as private is hard because they are afraid 
to be criticized. Students, like everybody else, worry about what impression they make on others and each 
element of the Story is carefully selected and organized to disclose what they want. The DS process enables 
students to undergo a process of self-reflection on who they are and what they wanted to show, whether they 
then disclosed their thought or not. 

Additionally, DS is emotional, sometimes upsetting. However, the shift to personal perspective from 
which emotion stems is associated with higher-order cognition, positive student development and 
personalized, closer and less formal learning. Moreover and connected to emotion and self-disclosure, 
interpersonal relationships influence have significant impact not only at the personal level, but also on the 
academic and the professional realms as well. However, we would like to assert that while these three 
perspectives are intertwined and cannot be dissociated, our practical experience as teachers has demonstrated, 
the personal is still seen as unessential and even uncalled for in HE by teachers and students alike. Students 
are understandably reluctant to talk about themselves and what they perceive to be as private and not 
belonging to the field of academia. Teachers seem to have the same opinion, admitting that there is an 
invisible boundary that is not crossed unless students volunteer the more personal details. This raises the 
question of what is considered appropriate in HE, what is perceived as private, and what is considered public. 
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ABSTRACT 

Despite the increasing attention given to design in business. Design Thinking has had little impact on the quality of 
business school education. Building upon the foundations of long-standing critiques of management education and the 
potential for student-centric learning, the authors propose that the use of Design in Practice can significantly improve the 
learning experience. The contribution concludes with an outline of the tenets of this vision that underpin their current 
work in management and corporate education. 


KEYWORDS 

Case Studies, Design Thinking, Management Education, Learning Technologies, Pedagogy 


1. INTRODUCTION 

In the following contribution, we argue that Design in Practice offers a fundamentally different approach to 
improving the quality of business school education. We begin our discussion with a quick review of the 
traditional critiques of business education. We then explore the relevant tenets of design thinking to pinpoint 
the processes and the methods that can address these shortcomings. We conclude with an expose of our own 
practice in business schools, ExecED and corporate education. 

Business school programs continue to tout the "success" of innovative design oriented companies like 
Apple, Google and Whatsapp. 1 Certain programs insist upon the financial performance of these examples, 
others the nature of their products, and others still the quality of their leadership. Few apparently offer insight 
into how little their success might actually depend on applying the traditional principles of management. To 
what extent can design thinking offer business schools an alternative vision of management education? 

1.1 Pedagogical Challenges 

Various authors have well documented the long standing issues with the MBA in particular and management 
education in general. These challenges hinge as much on the pedagogical choices that have been made as the 
reluctance of business schools to adapt their programs to new market challenges. Let's review a number of 
these issues before evaluating the value propositions of design thinking. 

One principal issue with business school education is its reliance on case study methodology that favors 
the notion of “one best way". Markulis (1985) notes that traditional case studies are most often sterile, 
impersonal, outdated, and subject to instructor bias. Burns ( 1984) argues that case study methodology reflects 
normative prescriptions rather than the real-life conditions in which companies succeed or fail. Most 
industries and markets today are not characterized by clearly defined problems and ready-made solutions, but 
by challenges (declining profits, underemployment, engagement...) of which understanding the nature of the 
problem is the major hurdle. 

Another issue deals with the choice of business disciplines that focus on a limited number of analytical 
skills and competencies. The skills needed to address of multitude of business challenges, ranging from 
hyper-competition to dealing with mass personalization, are often missing from traditional programs. Bennis 


See for example The Tuck School's "Deconstructing Apple", Harvard's "The Online Economy", or IE's Masters in Management 
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and O'Toole (2005) claim that the focus of graduate business education has become increasingly 
shortsighted — and less and less relevant to practitioners. 

A third objection addresses the inability of most programs to account for the uncertainty that characterizes 
most markets today. Snowdon and Boone (2007) suggest that the majority of business problems that can be 
solved through the use of “best practices” have been, while the complexity of the problems that linger require 
different forms of decision-making. Whereas most business programs privilege inductive or deductive 
reasoning, neither is associated with the third level thinking associated with innovation. 

A fourth criticism revolves around the project work given in class and out which emphasizes an 
unrealistic view of teamwork. Roger Martin of the University of Toronto suggests, "We teach a very narrow 
form of collaboration, which is to find somebody who thinks like you and then work together" (Dunne et al, 
2006). This approach has little in common with corporate practice where physical meetings are expensive, 
time-consuming and often very difficult to arrange. In the modern workplace, managers are continually 
struggling with discontinuous time, competing on different agendas, and being evaluated on work 
accomplished outside the meeting room. 

In a similar vein, the notion of productivity has changed. Sinofsky (2013) suggests that traditional visions 
of management based on hierarchy, top-down decision making and strategic planning are dubious mirages in 
markets flattened by the presence of ubitiquous information, connectivity, and mobility. In this view, 
productivity can best be studied today in gauging a manager’s ability to act effectively upon real-time 
information. 

Finally, teaching methods favor analytics rather than practice. Students who lack the experience to 
properly analyze and contextualize working knowledge poorly digest the codified, abstract premises that 
provide the staple of most management education programs. Mintzberg (2004) argues vigorously that 
teaching MBA students best practices won't help them learn how to manage, ‘ Organizations are complex 
phenomena. Managing them is a difficult, nuanced business, requiring all sorts of tacit understanding that 
can only be gained in context ’. 

The introduction of learning technologies has done little to address these challenges. The progressive 
introduction of Learning Management Systems (LMS), MOOCs, and now mobile applications has filed to 
address the pedagogical challenges to management education. By simply mirroring existing courses and 
approaches, learning technologies have often amplified the challenges in providing more effective designs to 
learn about business. These additional challenges include: 

E-learning pushes students out of the classroom, but it doesn't remove the need to take into account the 
context in which students learn. Context itself is a shell - learning not only occurs in a context, it creates 
context through the qualities of interactions between students and their professional environments (Sharpies 
et al., 2007). In short, the pedagogical value of learning technologies isn't found in the applications 
themselves, but in how students, faculty and organizations use these technologies to engage with their 
professional communities. 

Attempts to improve digital technologies without accounting for the specific nature of higher education 
appear as ill-fated as efforts to improve the classroom by limiting or banning the use of mobile phones, 
tablets and personal computers in class. The two are inherently intertwined in the modern classroom - we 
can't keep telephones out of the classroom no more than we should keep the classroom out of technology. 

Many examples of learning technologies today try to mimic the conditions of either the classroom 
environment, or the workplace without taking into account the specific constraints that each environment 
imposes. The nature of both the work and learning places - the vision, the space, the participants, and the 
outcomes - go a long way to explaining the challenges of doing real work in the classroom, and effective 
continuous learning at work. The goal of technology might best be served by not mimicking either "place" 
but by providing a bridge between the two where real-life business challenges can be brought to school, and 
learning outcomes can be rapidly applied in the workplace. 

To date learning technologies have simply reproduced the inductive or deductive logics inherent in 
business education. Whether they integrated multiple choice exams or discussion around business cases, 
learning technologies have rarely helped elucidate the nature of wicked problems. As we have argued 
previously, one of the critical success factors for learning technologies are engaging the students, retaining 
their attention, motivating them to invest in the experience at hand, and encouraging the physical application 
or reproduction of targeted skills (Schlenker, 2014). 
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2. DESIGN IN PRACTICE 

Design thinking challenges the assumption of business as usual in order to create new connections (Brown, 
2009). Design thinking encompasses the mental processes that are commonly used to design products and 
services. The associated process begins by analyzing behaviors and motivations, and then integrates the 
technical, financial and commercial considerations that shape the life of a project. In management, design 
thinking is applied to project-based work that addresses complex or “wicked” problems (Dunne, 2006). 

The roots of Design Thinking can be traced back from classical concerns with participatory design that 
favor integrating use studies into project prototyping (Di Russo, 2012). In his work on “user-centered 
design, "Norman (1998) stresses the need to take into account user's objectives and motivations in "making 
things visible”. In The Sciences of the Artificial, Simon(1996) suggests that design is a process that aims to 
improve the value of artifices like products, services and systems. 

The Design Thinking process can be juxtaposed with the tenets of traditional management. Roger Martin 
(2004) suggests that the focus on classical management is in improving repetitive tasks, whereas the design 
approach models work on a project to project basis. The nature of work itself is most often classified within 
well-defined roles, where a design approach sees work inherently as a collaborative exercise. Management 
theories proposed problem-solving models that are either deductive or inductive, while design theorists 
privilege “abductive” approaches. 

Managers evaluate an activity’s importance by the size of its budget and its staff, whereas design thinking 
suggests that success is tied to unbundling wicked problems. Management practice uses constraints to define 
the scope of action, where, as designers see constraints as opportunities to redefine the scope of potential 
activity. Finally, and importantly, management education encourages students to focus on one best solution 
to a problem, whereas design theory seeks to encourage the development of the larger number of potential 
solutions to a problem. 

Simon (1996) goes on to describe seven activities in the design process: Define Research, Ideate, 
Prototype, Choose, Implement, and Learn. Although has been debate since around which activities are 
critical to this process, practitioners generally agree that Design Thinking requires defining the right problem 
to solve, creating and evaluating the different options, nurturing an environment conducive to 
experimentation, and building testing the proposed solutions in the real world environment. 

These same practitioners insist that these activities aim to develop specific skills sets in students and 
managers alike: the abilities to deal with ambiguity, to be curious, to develop holistic views of the problem, 
to develop empathy, to work collaboratively, and to maintain critical distance. As Waloszek(2012) 
concludes, Design Thinking can be understood as a “ methodology that combines empathy for the context of a 
problem, creativity in the generation of insights and solutions, and rationality and feedback to analyze and fit 
solutions to the context. “ 

Design Thinking has been introduced over the last two decades as a subject of study in a number of 
business education programs. The value proposition of design thinking however isn't in analyzing its impact 
on business but applying the concepts of this methodology in developing management education as a whole. 
Specific points in which Design Thinking can improve how students learn about business include: 

The student is inherently part of the problem that must be addressed. "Teaching the student" is less 
important in business education than helping future managers effectively address customer challenges. Each 
student brings unique motivations, experiences and skill sets to class. Most can't relate with the business 
context under study - their empathy and implication are pre-requisites in solving the problems at hand. 
Finally, success depends upon practice: students must practice what teachers preach. 

Business problems solving requires a much deeper understanding of the user and of the user experience 
than we do in business schools. One of the core ideas in this vision is that the people using the products and 
services are different from those who manufacture and implement them. User experience is about creating 
memorable and experiences that have meaning for the consumer. Design Thinking implies using quantitative 
and qualitative approaches to develop a better understanding of the data. 

Business challenges are a result of a system of structures, patterns and events, rather than just the events 
alone. Any system is a web of interrelationships between people, information and physical technologies. 
There is a need to understand the essential relationships operating at various levels of the system, as well 
different strengths and probabilities for change. 
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Design Thinking takes into account how context shapes both the problem and the potential for viable 
solutions. In Design Thinking a problem is not only defined by its operating context but also by the 
constraints imposed on the problem solver. The greater the constraints: the better the chances of producing 
truly innovative solutions. This approach postulates that constraints are a source of new ideas, and should be 
fully recognized as levers rather than inhibitors to creativity. The more constraints a problem solver is forced 
to confront the better the opportunity to break out of the box of previous experience to find innovative 
solutions to the problem. 

The logic inherent in Design Thinking can help students visualize solutions to complex problems that 
elude best practices. This "abductive" logic can be understood as "the process of forming an explanatory 
hypothesis." Charles Sanders Peirce studies of the origins of new ideas led him to believe that innovation is 
tied neither to inductive nor deductive reasoning, but to “logical leaps of the mind” when our observations 
don't quite existing frames and models. This form of modal reasoning called "abductive logic" explores what 
could potentially be true (Martin, 2009). 

Design Thinking insists on the necessity of formulating a large set potentially useful ideas, services and 
products, gradually improving their fit with the problems under study, prototyping, giving the product to the 
consumer and then improving it some more. The advantages of a prototype a product, or simulating a service, 
include producing a better result at a more reasonable cost and contributing to strengthening empathy and 
engagement with the organization. 

Table 1. The 3Ps of Design in Practice 
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Based on our own pedagogical experience inEurope and abroad, we believe that the effectiveness of 
Design Thinking can be enhanced by focusing on Design in Practice. This concept implies that the 
application of design approaches can be enhanced by drawing the necessarily parallels between design 
thinking and pedagogy. Critical considerations include: 

2.1 Constructing a Holistic Approach that Positions Each Student in Context 

Design in Practice, rather than ignoring the limits of the classroom and the students' experience, recognizes 
that business challenges and solutions are context dependent. With this in mind, instructors, like 
practitioners, need to elucidate the context: the places, processes and people in which class participants 
evolve. The participants themselves will benefit from exploring the relationship between the content being 
provided in each program and the context(s) in which it is distributed, analyzed, and discussed. As in real life 
experience, these considerations include geography, time, physical resources and budget. 
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2.2 Co-Designing a Learning Place that Fosters Student (and Instructor) 
Engagement 

The value proposition of Design Thinking is less in its approach to teaching management than in its 
application to the design of participatory learning "places" in which students, faculty and administration (or 
managers, facilitators and sponsors) take responsibility for transforming content into action. A learning place 
is constructed, both physically and digitally, around a vision of what the place represents, the events that are 
staged there, the participants, the learning outcomes and gateways between this place and the work place. 
Design in Practice suggests that classrooms, much like participants, are part of both part of the challenge and 
the solution of management education. 

2.3 Deploying Physical Technologies that Are Adapted to the Challenges to Be 
Addressed 

Much the same can be said for the physical technologies found in the modern classroom. How does the 
physical layout of the auditorium, workshop or seminar room facilitate or hinder the students' understanding 
of the context of the subject under study? What high and low tech tools are provided to encourage the 
student's implication and appropriation of the subject at hand? How does the course support documents 
mirror or differ from the information available in the real world? How do the school and the instructor build 
bridges between the classroom and the workplace to encourage the participants to apply the lessons learned? 

2.4 Recognizing the Ubiquitous Nature of Information 

Class content is formal and informal, structured and unstructured. Reading lists in class are similar to project 
briefs at work: both are necessary but often incomplete in understanding what needs to be known to tackle 
the business problems at hand. Design in Practice suggests that students should be stimulated to explore what 
unstructured information is available, on the Web, and from interviewing the business community. 
Instructors should be encouraged to explore how the students are accessing, filtering, aggregating, applying, 
and sharing information sources. Sharing information needs to go beyond working in small groups - 
exploring the use and meaning of information with end-users, students in other disciplines, and managers in 
the business community can be a critical source of cognitive dissonance. 

2.5 Developing Each Student's State of Mind 

Finally, Design in Practice suggests that the ultimate goal of management education is not "spreading the 
good word" but helping the students transform data and information into managerial action. Our vision of 
Design in Practice implies that the learning outcomes depend upon helping each student manager understand 
how they use information to develop their managerial capabilities. How are the students framing the 
problems to be solved? How are they encouraged to analyze the visible constraints to fuel new ideas and 
potentially new products, services or systems? How does the specific pedagogy, and the program as a whole, 
represent a call to action? 


3. CASE STUDIES 

3.1 MBA France-India - Management Innovation 

We have been applying the principles of Design in Practice to in the MBA France-India program in 
encouraging students to improve significantly the impact of the learning "place". One of the major aims of 
this MBA, which draws students principally from southwest France and the Karnataka region, is to provide 
accessible managerial skills to work cross-culturally. The objectives of the management innovation module 
were to introduce the students to the various forms of innovation, to elucidate varying practices in fields 
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ranging from social commerce, omnichannel distribution to digital transformation, and to encourage the 
students to apply the lessons taught in small group projects. 

A number of constraints shaped the project. To begin with, the diverse backgrounds of the students: 
engineering, technology, as well as the social sciences and the humanities, pleaded in favor of an integrated 
multi-disciplinary approach to innovation. Second, the program's spatial distribution - one-third of the MBA 
is run in Pau, one-third in the Bangalore region, and one-third in internships in international companies - 
hinders the students' identification with a host school. Over the years, the need to build a stronger group 
identity and the deeper implication of the participants in the program have been constantly underlined the 
program's staff. Finally, the majority of the students are constantly seeking to get out of class to practice 
management. 

We incorporated the concepts of Design in Practice into a semester project called Design your School. In 
this crowd sourcing project, students were encouraged to use apply the concepts of management innovation 
in redesigning learning places outside the traditional classroom, notably in the both the Commons and the 
Resource Library. The students were challenged to shape their "learning place" using physical resources, 
information technology and change management. The students were invited to anchor their vision in the 
current students' interests and motivations, and then to redesign a space around a specific vision, use 
scenarios, events, and desired outcomes. 

The preliminary outcomes of the project include several dozen student proposals from class participants, 
as well as a number of professors and students from others schools, on visions ranging from Feng Shui 
working environments to finance and distribution test labs to an innovation factories. Student participation 
proved markedly better than in many modules of the program; many students continued to pitch and improve 
their projects throughout the year. The module materials, as well as the students' projects have been 
incorporated into an interactive e-book that the students can comment and structure for their personal needs. 

3.2 FBS - A Multi-Disciplinary Foundation for Management Study 

We have developed the principles of Design in Process in proposing a new approach to learning about 
management at France Business School (FBS). The merger of four business schools in 2012 gave birth to 
both FBS and a vision of management education based on a cross-functional management program. Behind 
this vision, the school has sought to appeal to a wide range of students by promoting innovation and 
entrepreneurship. The key to the program is a first semester agenda based on sharing and knowledge transfer 
among the different disciplines. 

There have been several challenges in creating the conditions for Design in Practice. To begin with, the 
pedagogy needed to be restructured to solicit novel ideas, embrace challenges, and produce meaningful 
solutions for business. Course work had to be integrated in pedagogical processes that promoted 
collaborative work and prototyping. The traditional classrooms have given way to "hotspots" integrating co- 
working spaces and digital learning technologies (MOOCs, an LMS and broadband internet access). The 
instructors have been encouraged to become "knowledge brokers" opening gateways to real-life experiences. 

Today, first year students are challenged to find fresh out-of-the-box solutions to today's major business 
problems. The "wicked" problems" they face are designed to strengthen their mental agility and develop their 
cognitive abilities for abductive reasoning. In class and out, students can write on the walls, build their own 
collaborative workspaces, and practice thinking-by-doing techniques (visual thinking, mocking-up, 
sketching, etc.). The student experience is based on an environment where problem solving, prototyping, and 
testing products, services and ideas have become the staple of the pedagogical process. 

Student evaluations indicate that Design in Practice develops twofold awareness: self-awareness on their 
capabilities to act as designers in producing ideas and solutions, and awareness on the power of collaborative 
work. The new program has accompanied a number of students in the creation of start-ups to put their ideas 
into practice both inside than outside the business school. Current students and the new alumni appear to 
create a cohesive community based on discovery and collaboration in line with the school's strategic vision. 


192 



1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


3.3 Microsoft - The OAM on Boarding Guide 

In designing Microsoft's OAM On boarding Guide, we are working with the corporation to help its 
employees structure and react to real-time flows of data, information and content. Microsoft, as one of the 
world's leading software companies, employees over three hundred employees to handle the logistics of 
service and delivery in four regional operations centers on four continents. The corporation's strategic shift 
from selling software licenses to promoting software and devices focuses particular light on operations 
management as the employees must quickly absorb new knowledge and deploy new skills. 

This shift in strategy has brought about a number of challenges in training new hires for the future 
challenges of software plus devices. The "one best way" to manage the new division does not exist as the 
corporation is exploring new markets. The operations managers are constantly on the go with little time to 
spend on classical classroom instruction. Communication between the different regional operations centers 
and between the OAMs themselves has been notoriously poor over the years. Finally, given the history of 
the company's success with desktop applications, the division has little experience with either mobile 
applications or mobile training. 

The Interactive On boarding Guide was conceived as an exercise in Design in Practice to address each of 
these issues. The texts themselves are delivered in the form of an interactive, social book that is updated each 
time the operations manager consults his Windows 8 tablet or mobile phone. Each page, each idea and each 
theme can be like, annotated and shared by each manager, who can also consult the comments of his or her 
colleagues in real time. The content is available off line, permitting each manager to consult the book 
anytime and anywhere. Natural language search allows the reader to quickly locate key ideas from anywhere 
in the text. 

The On boarding Guide is currently being rolled out worldwide. The Guide represents one of the 
corporation's first attempts to design and implement an application designed for a mobile work force. The 
ability to take notes and share ideas inside the book offers employees a real-time tool to improve horizontal 
communication between Oams and between the Operations Centers. The ability to update the texts in quasi - 
real time offers the division the possibility to update its vision as the market challenges of software and 
devices evolve. Although it is too early to have any quantitative analysis of use of the application, 
preliminary feedback from the beta test team has been markedly positive. 


4. CONCLUSION 

In conclusion, we have put forward a proposal to take a fundamentally different approach to business school 
education. Our argument has been built upon a foundation of common critiques of the MBA: the bias of case 
study methodology, the mismatch between management theory and the practice, the lack of attention given to 
how most markets and industries are evolving. Our proposal is built upon the precepts of Design Thinking - 
understanding that the students are both part of the challenge and an integral part of the solution, focusing on 
abductive methods for solving market challenges, and dealing with both ambiguity and complexity. 

Our vision goes beyond design thinking in suggesting that the value proposition for business schools is 
not in teaching the methodology, but in applying the approach to remodel management education. The scope 
of this effort cannot be limited to improving cognitive approaches, but instead extended to designing learning 
places, flows of information, and mindsets that support how students learn about management. Places, 
Processes, and People are interdependent considerations in pedagogy that inherently influence the quality of 
education. We conclude our argument with three examples of how we are putting Design in Practice. 
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ABSTRACT 

Despite the expansion of online and blended learning, as well as open education, little research has been undertaken on 
what motivates students to enrol inparticular study modes at university level. This project addresses this gap in higher 
education research by exploring the reasons why humanities students choose to study through specific modes. The 
research was conducted between October 2013 and March 2014 administering three waves of data collection to over 700 
students who were enrolled in humanities units being offered simultaneously through three different modes: on-campus, 
distance, and open and online. The findings suggest that students choose different enrolment modes based on factors such 
as personal, learning support, environment.advise and marketing, teaching and learning as well as logistics. However, the 
importance students ascribe to particular factors changes during their educational experience. This study found significant 
differences in the importance of factors betweeninitial and subsequent choices of enrolment mode, suggesting that the 
‘lived’ experience of students at university influences their perception of which factors are important. 
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1. INTRODUCTION 

Reductions to government funding of tertiary institutions have fostered a more competitive environment in 
Australian higher education over the past decadelde Zilwa, 2010; Bradley, 2008; Currie, 2002). Universities 
have been pressured and encouraged to explore new avenues for additional income and to view students as 
more consumer-like in their choice of a course and university. A heightened competition for prospective 
students has required tertiary institutions to adopt more flexible modes of delivering education to meet 
student demands. Pedagogically, blended models of learning combining face-to face and online experiences 
(e.g., lectures and tutorials supported by podcasts, online discussion, materials and activities) have led to 
teaching methods and resources that blur the boundaries of delivery! Lefoe and Albury, 2004; Lefoe and 
Hedberg, 2006; Woo et al., 2008; Aspden and Helm, 2004). Research has shown that online learning also 
increases the accessibility of tertiary education due to its capacity to overcome the spatial and temporal 
limitations of traditional teaching settings(Bates, 2005). Open access online education! Greenland and Moore, 
2014) and distance learning (Cohen, 2003)have thus become critical long-term strategies of many universities 
to encourage higher education participation (Allen and Seaman, 2006; Ziguras and McBurnie, 2011). Further, 
the implementation of web-based learning technologies across modes of delivery has blurred the distinction 
between the experiences of on- and off-campus students. Recent research has therefore emphasised the need 
to develop a validated measure for differences in the motivation to enrol in a specific course format( Johnson 
et al., 2013). 

There is a wealth of research on why students choose their institution of higher education. The reputation 
of the institution has been found to be the primary factor that guides students’ decision-making! Harkera et 
al., 2001; Chapman, 1981; Hayes, 1989). Also of importance is the reputation and “nature” of the particular 
course, the quality of teaching in the department, the department’s reputation and the friendliness of the 
department(Cebula and Lopes, 1982; Booth, 1997). However, there has been little research undertaken on 
what motivates students to study certain units through particular modes at university. Research on, and 
discussion of, why students study individual units in the humanities is largely absent from the literature. This 
project thus addresses a gap in the higher education research by exploring the reasons students choose to 
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study specific modes of study well asproviding invaluable information for shaping study programs, thus 
laying the ground for future research into educational pathways. 

1.1 Choosing a Study Mode 

The decision-making processes of students at the entry point to higher education have become the focus of 
several research projects. Studies conducted since the early 1990s have examined intrinsic motivations such 
as interest in an area of knowledge and related career opportunities(Sugahara et al., 2008; James, 2000). 
Bornholt et al. (2004)have shown how interlinking personal and social factors influence student preferences, 
confirming the results of an earlier study by James et al. ( 1999)that rural and remote locations and low socio- 
economic background impact on students’ choice to consider a higher education pathway in Australia. 
Demographic factors have been found to play a role in the choice of study mode: gender enrolment trends 
indicate that a significantly higher proportion of women than men choose online courses as an educational 
pathway to obtain a degree(Price, 2006). Moore and Kearskey (2005)have observed that the majority of 
students enrolled in distance education programs are adult learners between the ages of 25 and 50 years. 
There is also evidence that age, gender, educational background, work commitments and family status impact 
on completion rates in higher education(Colorado and Eberle, 2010; Tsay et al., 2000). However, studies into 
how these variables affect students’ preferences for a specific educational pathway that suits their learning 
and personal needs as they progress through their studies are yet to be forthcoming. 

1.2 Research Question and Hypotheses 

The present study was set to investigate and compare why students choose different modes of study, i.e., on- 
campus (mainly face-to-face study), distance (off-campus study including on-campus components), or open 
and online (open entry and fully online study). Previous research shows that there are many reasons why 
students choose different modes of study, for example: to fit with family and lifestyle priorities; balance work 
and study; an inability to get to campus due to distance or inconvenience; administrative and organisational 
constraints (e.g., timetabling of classes; special access and learning needs); or in response to differing 
pedagogical approaches, resources and services(Hrastinski and Jaldemark, 2012). In particular, the following 
research question and hypotheses were addressed: 

Do the importance of personal, logistics, teaching and learning, support, environment as well as advice 
andmarketing factors among students enrolled in different study modes (on-campus, distance, open and 
online) change from initial choice of study (not completed a level 100 unit) to the current choice of study 
(after completing one or more units)? We assume that the importance of factors (personal [Hi], logistics fH 2 ], 
teaching and learning [H 3 ], support [H4], environment [H 5 ], advice and marketing [H 6 ]) change from initial to 
current choice of study among students enrolled in different study modes (on-campus, distance, open and 
online). 


2. METHOD 

2.1 Setting and Context 

Macquarie University has a long history of providing flexible offerings and pathways to study. It has a well- 
established distance program to complement many of its on-campus offerings and several alternate pathways 
for entry to academic programs including the Jubilee Scheme, Non-award pathways and open and online 
programs through Open Universities Australia (OUA; www.open.edu.au). OUA offerings have increased in 
recent years with a total of 138 individual units now on offer at undergraduate, postgraduate and non-award 
level. At the undergraduate level, Macquarie has 92 individual units on offer as well as a Bachelor of Arts. 
Whilst not all units and programs at Macquarie offer the full range of flexible offerings and pathways, there 
are those (particularly in the Faculty of Arts) where students are able to choose between one of three modes 
of enrolment: 
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• On-campus mode: On-campus offerings, with the expectation of an on-campus presence and 
typified by a blend of face-to-face and online learning experiences (variable: on-campus study 
mode). 

• Distance mode: Equated to distance learning where students study off-campus however there may 
be an on-campus component. Delivery can vary from fully online to a blend of online, print and 
multimedia (variable: distance study mode). 

• Open and online mode: Open-access study and fully online delivery (variable: open and online study 
mode). 

The principle means of data collection was an online survey which was conducted between October 2013 
and March 2014 in three waves of data collection over three teaching sessions - Session 2, 2013, Session 3 
over the summer break and Session 1, 2014. These aligned with OUA’s third and fourth study period, 2013, 
and first study period, 2014. 

2.2 Participants 

Participants in the study were drawn from students studying units in the Faculty of Arts BA program that 
were offered concurrently in the three modes: on-campus, distance as well as open and online.In total, N = 
744 students from the Faculty of Arts participated(70% female, 29% male, 1% 
indeterminate/intersex/unspecified). Their average age was 27.16 years ( SD = 10.42). 7% reported having a 
physical or learning disability that impacted their experience at university. 77% were full-time students and 
23% were part-time students. 2% of the participants reported to be Aboriginal or Torres Strait Islander. 10% 
were in full-time employment, 19% worked less than 10 hours per week, 22% worked between 11 and 20 
hours per week, 12% worked 20 or more hours per week, and 37% were not in paid employment. Over half 
of the participants reported having completed the final year of secondary education (59%), 7% had completed 
a Diploma or Associate Degree, 13% had started and 3% completed a Bachelors degree, 1% a Masters 
degree, and 2% had completed a Postgraduate degree. The remaining participants reported to have 
VET/TAFE or other post school qualifications (13%), and 2% reported having no prior educational 
attainment. Initially 57.4% of participants were enrolled in on-campus mode, 5.4% in distance mode and 
22.5% in open and online mode. After studying for one semester or more there has been an overall shift in 
the enrolment pattern which is reflected in the current enrolment status of 51.2% in on-campus mode, 10.5% 
in distance mode and 18.6% in open and online mode. In each of the three survey waves, as an incentive 
participating students were offered the chance to win one of twenty $30 iTunes vouchers. 

2.3 Instrument 

The survey consisted of the following sections: 1. Enrolment profile, 2. Motivation to study, 3. Factors 
influencing initial choice of study, 4. Factors influencing the current choice of study, 5. Technology skills, 6. 
Demographics. Most items were answered on a five-point Fikert scale (5 = extremely important; 4 = very 
important; 3 = neither important nor unimportant; 2 = very unimportant; 1= not at all important). Table 1 
provides a summary of the factors covered in each of the sections. Items were adapted from a pilot survey 
conducted at Open Universities Australia and were supplemented by additional items generated by an expert 
team. The six factors have been successfully tested for reliability with Cronbach’s alpha .681 <r < .869. The 
survey was implemented on the Qualtrics platform (www.qualtrics.com). It took approximately 15 minutes to 
complete the survey. 

2.4 Data Collection and Analysis 

Using enrolment lists for units offered concurrently in the three modes (albeit with variances in start and end- 
dates), participants were invited to complete the survey using the Qualtrics platform bulk email function. All 
data stored on the Qualtrics platform was anonymised, exported, and analysed using SPSS V.21. Initial data 
checks showed that the distributions of ratings and scores satisfied the assumptions underlying the analysis 
procedures. All effects were assessed at the .05 level. 
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Table 1. Example items of the survey 


Section 

Enrolment profile 
Motivation to study 


Initial choice of study 


Current choice of study 
Technology skills 


Demographics 


Example items 

Program, major, units completed, study mode (on-campus, distance, open 
and online) 

How important are the following factors for undertaking university level 
studies? 

Factors comprised: To gain employment, To progress my business, To 
start a business, Career change, Career progression. Job requirement. 
Knowledge and skill development. Prerequisite to another course, and 
For personal interest. 

How important were the following personal factors in making your initial 
choice of study? 

There were six factors comprised of several items. 

1. Personal factors comprised five items: Personal, Cultural and/or 
religious orientation. Special/ specific learning needs, Confidence in 
your academic ability or capacity to succeed at university-level 
study and Prior experience of studying at MQ or OUA. 

2. Logistics comprised eight items: Cost of study per unit. Distance 
from campus, Ease of access to campus (e.g. transport, parking), 
Flexibility in studying at your own pace. Flexibility in studying at 
your time of choosing, Flexibility in managing work-life-study 
balance. Flexibility in studying at other universities and Range of 
units available to choose from. 

3. Teaching and learning comprised six items: Extent to which 
teaching and learning is conducted online. Being able to work 
collaboratively with other students, Access to study materials and 
resources, Expected workload. Engagement with academic staff, and 
Reputation of high quality teaching. 

4. Support for learning comprised four items: Access to services to 
support learning (e.g., writing, numeracy and literacy support). 
Access to IT services and support. Access to course and careers 
advice and Ease of administration. 

5. Environment and campus/community wellbeing comprised three 
items: Access to personal support services (e.g. medical, disability 
services counselling). Access to campus facilities (e.g. gym, 
swimming pool, clubs) and Meeting and socialising with other 
students. 

6. Advice and marketing comprised three items: Experience of other 
students. Advertising/ Website/ Social Media and Advice from 
Student Advisors or other university services. 

As for initial choice of study. 

Please rate your experience with using technologies for learning. 
Experience included: using the computer; surfing the Internet, with using 
blogs, wikis, podcasts, YouTube and discussion forums; doing Internet 
searches, setting bookmarks; uploading and downloading files, doing 
Internet searches; installing software and changing configuration settings 
on my computer; and getting help if I have computer problems. 
Socio-demographic information 


3. RESULTS 

In order to test our hypotheses, six repeated-measure MANOVAs with the importance of factors (personal 
[Hi], logistics [H 2 ], teaching and learning [H 3 ], support [H 4 ], environment [H 5 ], advice and marketing 
[H 6 ]) at two measurement points (initial and current) as a within-subjects factor, and study mode (on-campus, 
distance, open and online) as a between -subjects factor were computed (see Table 2 for descriptive statistics 
and Figures 1-3 for a visual representation of trends). 
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Table 2. Means, standard deviations of importance of factors for two measurement points and study modes 



Mea surement 

Study mode 

Factor 

point 

On-campus 

Distance 

Open and 
online 

Personal 

Initial 

3.08(.58) 

2.89(.73) 

2.88(.78) 


Current 

3.08(.62) 

3.10(.76) 

2.99(.76) 

Logistics 

Initial 

3.32(.77) 

3.52(.70) 

3.69(.59) 


Current 

3.29074) 

3.56(.70) 

3.50(.73) 

Teaching and learning 

Initial 

3.42(.90) 

3.73(.77) 

3.81070) 


Current 

3.23(.98) 

3.73(.85) 

3.70084) 

Support 

Initial 

3.3K.85) 

3.23(.94) 

3.26(.94) 


Current 

3.25(.82) 

3.08(1.09) 

3.02(1.03) 

Environment 

Initial 

3.28(.82) 

2.55(1.14) 

1.91(.94) 


Current 

3.29(,76) 

2.44(1.21) 

1.97097) 

Advice and marketing 

Initial 

3.27(.77) 

2.77(1.06) 

2.74(1.09) 


Current 

3.29(.72) 

2.67(1.21) 

2.67(1.14) 


Note . Means are based on a 5 point scale where 5 = extremely important and 1 = not at all important 


For importance of personal factor j H , /, the difference between measurements (initial and current) was 
significant, F{ 1, 741) = 18.45, p< .001, if = .024 (small effect). We also found a significant interaction (time 
and study mode), F( 2, 741) = 6.78, p = .001, if = .018 (small effect) and a significant difference between 
study modes (on-campus, distance, open and online), F( 2, 741 ) = 3.05, p= .048, if = .008 (small effect). 

For importance of logistics factorf H 2 ], the difference between measurements was significant, F{ 1, 741) = 
4.10, p= .043, if = .005 (small effect). We also found a significant interaction (time and study mode), F( 2, 
741) = 4.64, p = .010, if = .012 (small effect) and a significant difference between study modes (on-campus, 
distance, open and online), F( 2, 741) = 11.08, p< .001, if = .029 (small effect). 

For importance of teaching and learning factorf If], the difference between measurements was 
significant, F( 1, 741) = 8.03, p = .005, if = .011 (small effect). However, no significant interaction, F( 2, 741) 
= 2.96, p = n.s. was found. However, MANOVA revealed a significant difference between study modes (on- 
campus, distance, open and online), F( 2, 741 ) = 17.72, p< .001, if = .046 (small effect). 

For importance of support factorf H 4 ], the difference between measurements was significant, F{ 1, 741) = 
26.20, p< .001, if = .034 (small effect). We also found a significant interaction, F( 2, 741 ) = 4.98, p= .007, if 
= .005 (small effect). The difference between study modes was not significant, F( 2, 741) = 1.81, p = n.s. 

For importance of environment factorf H 5 ], no significant difference between measurements,/ 7 ) 1, 741) = 
.21, p= n.s., and no significant interaction, F(2, 741) = 2.19, p = n.s., was found. However, MANOVA 
revealed a significant difference between study modes (on-campus, distance, open and online), F( 2, 741) = 
144.71, p< .001, if = .281 (strong effect). 

For importance of advice and marketing factorf H 6 ] , no significant difference between measurements, /•’( I , 
741) = 2.61, p = n.s., and no significant interaction, F( 2, 741) = 1.91, p = n.s., was found. However, 
MANOVA revealed a significant difference between study modes (on-campus, distance, open and online), 
F{ 2, 741 ) = 35.25, p< .001, if = .087 (small effect). 

To sum up, the different between study modes for the environment factor showed the biggest effect size 
(if = .281) indicating high importance of environmental factors for on-campus students and low importance 
for open and online students. Small effect sizes were found from initial to current mode of study for personal, 
logistics, teaching and learning, as well as support factors. Therefore, we accept the following hypotheses: 
//,. //,. If. ll 4 . 
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Figure 1. Trajectory of mean of importance of factors for on-campus mode of study 



Figure 2. Trajectory of mean of importance of factors for distance mode of study 



Figure 3. Trajectory of mean of importance of factors for open and online mode of study 
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4. DISCUSSION AND CONCLUSION 

The study explored the influence of six factors (personal, logistics, teaching and learning, support, 
environment, advice and marketing) on the choices students are making about their mode of study, whether 
in on-campus and distance mode at Macquarie University or in open and online mode through Open 
Universities Australia. Wehypothesised that the importance of these factors for students would change as 
they progressed through their studies. We found that there were changes in importance for all factors, and 
that this was variable depending on study mode. There were significant changes from initial to current study 
mode in thepersonal, logistics, teaching and learning, and support factors, while the environment as well as 
advice and marketing factors were relatively stable. 

The high mean score of importance for the teaching and learning factor for initial and current mode of 
study across all cohorts emphasises the importance of maintaining and ensuring quality delivery and 
providing a high level of transparency in the information provided to the students about the nature and 
requirements of different modes of study, teaching methods, the learning experience, workload and 
assessment processes. That the mean was higher on this factor for the distance as well as open and online 
cohorts when compared with the on-campus cohort, suggests that an absence, or relatively limited degree, of 
face-to-face contact serves to increase the importance of such items for students. 

At Macquarie University there is consistency in the design of learning across the three different modes, 
with students reporting an expectation that technologies and online learning will be integral to their 
university experience (Gosper et al., 2013). The LMS and web-based lecture recordings, both of which are 
mandatory in the Faculty of Arts are key components of a blended environment for on -campus students and 
they also form the backbone for the delivery in distance mode as well as open and online mode. The key 
characteristics which define a distance learning environment - interactions between student-content, student- 
student and student-staff (Anderson, 2003) - are virtually the same for distance as well as open and online 
students, and to a lesser degree on -campus students, thus blurring the boundaries between the three modes of 
delivery (Woo et al., 2008). This helps to explain the shared high level of importance for the teaching and 
learning factor. 

Differences in the study environment, though, explain the similar means and trajectories for the distance 
and open and online cohorts, and their distinction from the on-campus cohort. The environment factor 
captures this distinction most clearly with the later cohort according it a significantly higher mean importance 
than both the distance as well as open and online cohorts. That distance students in turn rate this factor higher 
than open and online students reflects the former’s closer association with the Macquarie campus. Distance 
students have an equivalent access to campus and library facilities as on-campus students if they choose to 
use them. Open and online students, however, are restricted in this usage and all their learning is conducted 
online. For on-campus students environmental factors did not significantly change in importance after their 
initial choice, suggesting that as they become more aware of on-campus facilities and acclimatised to campus 
life, these become habitual components in their overall study experience. 

For some time now students in Australia have been demanding more flexibility to enable them to study 
while managing their work and family commitments (Mclnnis and Flartley, 2002). Backing this up, a study 
by the Australian Vice-Chancellor’s Committee (2007) found 71% of Australian university students 
undertake paid employment during semester, working an average of 15 hours per week. It is likely that in 
2013 the work-study balance was similar or even more demanding, which begins to explain the importance 
studentsaccord to the logistics factor. This factor ratessecond in importance for all students’ initialand current 
choices about study mode. Distance as well as open and online offerings can provide the flexibility to deal 
with the logistics of access, cost and travel(Bates, 2005), explaining the high mean scores registered for this 
factor by distance and open and online students when compared with on-campus students. Our results reveal 
that when comparing initial and current modes, there was an overall increase in distance enrolments from 
5.4% to 10.5% and at the same time there was a fall in internal mode from 57.4% to 51.2% as well as in open 
and online from 22.5% to 18.6%. The changes to distance and on-campus enrolment likely indicate the usage 
of the distance mode as a strategy to balance competing demands on time. This is facilitated by the ease of 
enrolment in either mode for internally enrolled Macquarie students. The drop in open and online enrolments 
suggests that some students may use OUA as an avenue into other higher education programs, although 
further research is required to map such practices. 
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The implications of these findings can be approached from two perspectives. The first is that given the 
universal importance of the teaching and learning, logistics andsupport factors.information about these 
factors should be transparent to students from the outset of their studies. This would have an impact on the 
style and content of marketing campaigns and also the advice given to students at orientation and enrolment 
sessions. Alternatively it could mean that a ‘lived experience’ is necessary before students are in a position to 
fully understand their preferred learning mode, as well as the support, environmental and logistical factors 
impacting their University studies! Dobozy and Ifenthaler, 2014). If this is the case then the implication^ for 
universities to enable students the flexibility to move more easily between different modes as they mature as 
learners and/or their circumstances change. Indeed 33% of participants expressed a desire for more flexibility 
in this area. 

This paper has reported onfindings examining the factors influencing students’ initial and subsequent 
decisions about study modes. The general trends are evident and provide useful insights that can be used by 
universities to attract, support and retain students in a competitive environment. Further analysis of data is 
currently being undertaken into the different elements within the six factors to provide a more nuanced 
understanding of the choices being made by students, as well as the impact of demographic characteristics on 
these decisions. In addition, the survey is limited by its quantitative nature and further qualitative research is 
planned to provide further interpretation of the trends that have emerged. Extending the research beyond a 
single degree program in one faculty will also provide a more comprehensive understanding of students and 
the choices they make. 
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ABSTRACT 

The No Child Left Behind (NCLB ) legislation spawned a plethora of standardized testing services for all the high stakes 
testing required by the law. We argue that one-size-fits all assessments disadvantage students who are English Language 
Learners, in the USA, as well as students with limited economic resources, special needs, and not reading on grade level. 
The SAVE Science project was developed to explore whether and how contextually driven assessments support these 
students demonstrate their understanding of science content in middle grades in the USA. Preliminary findings from this 
6-year study suggest that situating assessment in virtual contexts does in fact help students in answering multiple choice 
questions correctly and also helps students better understand their own science knowledge and learning process. 


KEYWORDS 

Standardized testing, virtual environments, critical theory. 


1. INTRODUCTION 

The ubiquitous standardized tests developed by a small number of educational services companies and used 
in American public schools contain questions culled from the cultural experiences of, and based on the 
language abilities of, the test content developers. All students are expected to be familiar with this content, 
but in truth it is often far removed from the experiences and skills of actual students. And this expectation 
automatically disadvantages groups of students, particularly English Language Learners in the USA, students 
with limited economic resources (which can constrain exposure to varied cultural experiences) and students 
with special needs. Students taking such tests experience an existential dislocation: they must answer 
questions in a formal and rigid way, questions that may call for cultural acuity or information they may not 
have, and questions written by unseen experts for whom this information is often intuitive. This can turn test 
taking into an Escher-esque endless loop of disconnectedness. 

A Principal of a K-5 public school in New York penned an op-ed in the New York Times recently which 
touched on that disconnection, noting that English Language Arts standardized test content (developed by 
Pearson Publishing for the State of New York) presented students with questions that were “confusing” and 
“developmentally inappropriate. . .There was a strong emphasis on questions addressing the structure rather 
than the meaning of texts. There was also a striking lack of passages with an urban setting.” (Phillips 2014) 
These tests carry high stakes for both the schools and the teachers. This shift to high stakes testing is yet 
another national educational policy change that, as educational researcher Michael Apple notes, is an 
outcome of ongoing cultural and political conflicts at the macro level. (Apple 2007, p. 165) At the classroom 
level, in order to make these tests “count,” teachers must work within a new and permanent professional 
contradiction — they are trained to differentiate instruction to meet the needs of students where they are but 
they then must standardize testing. 

Standardization has been embraced by school reformers and educational policy makers in the USA as a 
means of tracking the performance and accountability of schools, teachers, and students alike, which leaves 
the ethical educator with few options. To counter the uniform application of these standards of knowledge to 
students who have varying skills, experiences, and language abilities, some refuse to give such tests, as a 
cluster of teachers in the city of Seattle, Washington did in 2013. Some have actively protested the test 
content and the standards aligned with them, most recently a group of teachers, parents, and administrators in 
New York City. And still others have worked within the standardization framework to create test 
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environments and test content that minimizes that endless loop. One goal of the SAVE Science study is to 
address this problem in testing and lessen that disconnect between test content and student experience 
through the creation of a new kind of assessment tool for middle school science students, where tests are 
taken by navigating virtual game environments and students use visual cues and inquiry skills to solve 
contextual problems. These assessments are proximal tests, directly linked to curricular concepts but 
delivered in a new context, but also incorporate test content derived from distal measures - in this case, the 
statewide Pennsylvania System of School Assessment (PSSA) tests. (Geier, et al 2008, p. 923) This paper 
presents a critical theory view of the importance of the SAVE Science project and what it brings to the 
dialogue around high stakes testing and differentiated instruction/learning. 


2. SAVE SCIENCE 

The SAVE Science study is an NSF-funded project to create, implement, and evaluate computer-based 
assessment modules for science content in the middle grades. The modules are designed to enable students at 
varying skill levels and language abilities to perform a series of problem-solving tasks in a virtual world, 
tasks that provide data about how those students apply content knowledge related to classroom curricula. 
These alternative assessments address several of the conditions needed for better science assessments, chief 
among them contextualization. Because students have a difficult time applying their understandings of 
science content and their own experiences to the decontextualized questions found on multiple choice written 
tests, assessments should contextualize questions. One example of this, from a recent PSSA test, is a question 
about freshwater fish and how they adapt to live in weedy areas of lakes, a question that urban students may 
have to answer with no lived experience to draw from. A SAVE Science test module would provide the 
student with context by offering an immersive virtual environment (IVE) where this question was 
accompanied by an active rendering of that type of fish swimming in its environment, and students could 
observe the fish in its habitat before answering the question. Another example: the SAVE Science module 
aligned with content about gas laws provides students with an IVE where basketballs are played with an 
indoor basketball gym, and then are used on an outdoor basketball court in cold weather, (see Ketelhut, et al 
2013 for more detail) Students must determine why the basketballs bounce differently between inside and 
outside by gathering empirical data, analyzing that data, and reporting back to the appropriate character their 
hypothesis and evidence. These activities are then followed by 3 standardized test items that correlate with 
the high stakes test questions about the same content. 

Evidence centered design principles (Mislevy, 2011) were used to develop each of the SAVE Science 
modules. As Mislevy noted, “One challenge is that the development of a valid simulation-[virtual-] based 
assessment requires the expertise from disparate domains come together to serve the assessment’s purpose 
(typically including subject matter knowledge, software design, psychometrics, assessment design, and 
pedagogical knowledge).” The SAVE Science team was comprised of 1) an expert in science content, science 
teaching, and assessment design for science content; 2) an expert in designing virtual environments for 
assessment; and 3) a psychometrician. This team, along with 12 science teachers, four science education 
doctoral students, one science education post-doctoral fellow, and one qualitative research specialist, 
designed five assessment modules (2 for 7 th grade, 3 for 8 th grade), plus two introductory modules (1 for 7 th 
grade, 1 for 8 th grade). 

Working with two senior science teacher leaders from a large urban school district, the team identified 
specific areas of middle grade science curriculum that was determined to be difficult to assess through the 
high stakes objective assessments. Given these assessments are objective in nature requiring reading of 
English (in the U.S.) on grade level, evidence centered design allowed the designers to create a virtual world 
where the students were put into a context (e.g., a basketball course inside and outside) to gather data around 
a problem presented as urgent, but within the context recently taught in the curriculum. Using Vygotsky’s 
(1978) zone of proximal development theory, each assessment module was designed to be just beyond the 
capabilities of the students, but close enough to not be too complex. The evidence gathered by the system 
included a trace of every non-player character encountered, data gathered using the science tools built into 
each module, and a 3-dimensional time-stamped map of each student’s movements within the virtual 
environment module. In the end, students answered both objective questions and open-ended explanations of 
evidence, and giving a solution to the original problem/question posed. 
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The results are a rich set of data that can be used to determine whether the student understood the 
question posed and data collected sufficiently to successfully respond to the assessment questions at the end 
of the module. But analyzing these disparate bits is not simple. Most teachers are not taught data-driven 
decision-making using multiple data points/sets. Evidence from SAVE Science initially suggests it is possible 
to identify those students who clearly understand the science content in each module from those who clearly 
do not (Sil, Shelton, Ketelhut, & Yates, 2012). We continue to refine the analysis toward a prediction model. 


3. DISCUSSION 

These efforts are one direct attempt to reduce the disconnection traditional standardized tests can cause 
among students, which is an issue that permeates education in the accountability era. Today approximately 
21% of the public school population in the USA is made up of English Language Learners (ELL) and only 
3% of these students reached proficiency or above on the 2009 National Assessment of Educational Progress 
(NAEP) reading assessment, as compared to 35% of native English speaking students. (Lara-Alecio, et al 
2012, p. 987) One aspect of ELL students’ lack of proficiency on these tests is the “unnecessary linguistic 
complexity” of test items they encounter on such tests. (Abedi & Gandara 2006, p. 39) These students are 
directly disadvantaged by the test content they are required to master, test content developed by native 
English speaking test developers presented out of context and accompanied by minimal visual cues. Along 
those same lines, NCLB legislation ushered in an era of testing “focusing solely on student outcomes” as a 
means of improving schooling, but ELL students “disproportionately attend high-poverty schools with 
limited resources, and fewer schools offer bilingual education programs than did before the passage of 
NCLB.” (Menken 2010, p. 127) 

Recent research has shown that urban districts in the USA overall are suffering the consequences of 
accountability systems based on test scores, and that “academically disadvantaged students in large cities are 
currently being left behind because the use of proficiency counts in NCLB does not provide strong incentives 
for schools to direct more attention to them.” (Neal & Schazenbach 2010, p. 280) Compounding this issue is 
the fact that the AYP measure is intended to reflect a rising minimum threshold for improvement, so that 
“schools that begin with low test scores, typically urban schools with a high percentage of children living in 
poverty in the USA, can have improving test score results, but because they do not rise above the minimum 
threshold, remain classified as failing... [and] because NCLB requires that by 2014 essentially all students 
need to pass every test, almost all the schools in the USA will be found to be failing.” (Hursh 2013, p. 577) 
This conundrum reveals the limitations of, if not the fallacy of, an uncritical reliance on high stakes testing at 
the broadest levels. At the student level, research has shown that learning through a type of digital gaming 
similar to that developed by SAVE Science can be directly linked to learning outcomes, and that 
contextualized information in game environments allows us to “measure [students’] growth across time, and 
track different trajectories to mastery.” (Herold 2013) 

As noted above, teachers in the USA are inculcated with the concept of differentiating instruction based 
on student ability, including, but not limited to, English language proficiency, special needs, reading level, 
and prior learning. The age of accountability, symbolized by high stakes objective type tests which include 
extensive reading passages to convey science content, poses a conundrum for teachers: Do they follow 
ethical teaching and differentiate instruction, knowing that not all students will achieve the same level of 
proficiency or even basic knowledge, or do they teach to the test? SAVE Science was design to challenge the 
notion of what is a test in science, putting the questions in to context where the student can demonstrate 
through multiple ways their understanding of content and scientific inquiry. 


4. CONCLUSION 

Two practical goals of the SAVE Science study were to develop new types of computer-based assessments 
that integrated and contextualized science content for students and to enhance understanding of students’ use 
of inquiry processes in science through the use of such alternative tests. A key motivating question for the 
Principal Investigators of the project was “Can we create something that’s reliable and valid as an alternative 
to traditional testing?” And recent research supports the contention that assessments situated in virtual 
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environments can also offer insights into student understanding not easily captured with other assessment 
methods and provides information about students’ strategies in problem solving. (Ketelhut, 2007) So at its 
base level the SAVE Science study has been exploring how to improve students success in understanding and 
answering required test questions correctly. But the broader aim of the study has been about developing 
alternative forms of assessment which provide contextualized information for students and opportunities for 
them to demonstrate the ability to identify problems, collect data, and find solutions in a manner that does not 
alienate them or punish them for not intuiting knowledge that is not part of their lived experience. 
Preliminary findings from this 6-year study suggest that situating assessment in visual, virtual contexts does 
in fact help students in answering multiple choice questions correctly and also helps students better 
understand their own science knowledge and learning process. These promising results provide initial 
evidence that situating assessments in IVEs and situating test questions in context can play a role in 
improving standardized high stakes tests, and contribute to the ongoing conversation about what such tests 
measure, and how they are used. SAVE Science provides us with data about the use of virtual environment 
assessments with contextualized questions that may mitigate that endless loop of disconnection for students — 
and provides clear examples for test content developers to use in designing assessments that do a better job of 
meeting students where they are. 
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ABSTRACT 

The purpose of this paper is to provide a narrative of work in progress to validate a math app designed for number sense. 
To date I have conducted classroom research and pilot studies across ten early childhood classrooms in two schools and 
will begin an empirical study at the beginning of the 2014-2015 school year. Through my work I believe the fields of 
neuroscience, education, and digital science offer robust and unique ways to address at least two barriers I encountered: 
identifying instructional computer adaptive software containing embedded assessments and designed explicitly with 
cognitive models of learning; and developing ongoing collaborative research networks to validate this software. In an 
attempt to inform the work of those working in the fields of digital science, cognitive science and education, my 
reflection includes the background, content, context and observations of my studies to date, as well as insights and 
emerging hypotheses for consideration. 


KEYWORDS 

Computer adaptive software, collaborative research networks 


1. INTRODUCTION 

The quote in the title of this paper is from a former student; a testimony to being engaged while using a math 
app. Engagement is an important component of instruction to measure, but what methods exist for measuring 
learning and understanding of instruction delivered via apps? Does instructional software exist that allows for 
learning and assessment to occur simultaneously? How can classroom teachers undertake investigating these 
questions through formal research? Computer adaptive software offers great potential to solve these 
questions; collaborative networks offer the potential to research these questions. In this paper I offer a 
narrative of my work in progress to validate Native Numbers©, a math app, through collaboration with a 
network of researchers and educators. My purpose in sharing this narrative is to provide insights and 
observations that may inform future work of those in the fields of cognition, education, and computer 
science. This narrative adds to the limited body of research conducted explicitly through collaborations 
between educators, researchers and other stakeholders, in an attempt to validate learning effectiveness of 
educational software designed intentionally with cognitive models of learning. 


2. BACKGROUND 

As a teacher I must know if the academic content of any instructional resource is effective and as a 
practitioner -researcher I am bound to investigate effectiveness thorough formal or informal study. This study 
began during the last semester of my master’s graduate work. As background, I highlight here three 
watershed events as to why I chose to examine the educational effectiveness of Native Numbers ©, a math 
app for early number sense. First, moving from teaching prekindergarten to first grade I perceived many of 
my former students had not progressed in some of the math skills I considered they had mastered a full year 
previously. I had ultimate confidence in their kindergarten teachers’ instruction, so I was greatly puzzled. 
Second, my graduate work in the Mind. Brain and Education (MBE) program at the University of Texas, 
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Arlington (UTA) prepared me to take on the role of practitioner -researcher. Core to the program design is a 
goal of enabling teachers to become teacher -researchers, trained and ready to bridge the divide between 
neuroscience and education (Schwartz & Gerlach, 2011 ). Outside of my required course work, I read research 
and attended professional development related to number sense and experimented with different instructional 
methods. Third, while attending a workshop for Native Numbers ©, I recognized both the cognitive models 
and the number sense research this app had included in its design. I asked my director for permission to 
conduct a study of the app as an elective independent study. We contacted the developers and initiated a 
collaboration to study the educational effectiveness of Native Numbers © for early number sense. 

Validation of assessments and instructional resources for math is critical as the long term effect of 
mathematical difficulties, and the need for intense and explicit assessment and intervention, cannot be 
overstated. Morgan, Farkas, and Wu (2009) conducted a study using data from the U.S. Department of 
Education’s National Center for Educational Statistics to investigate live year growth trajectories of students 
with mathematical difficulties (MD). They found students who manifested MD in kindergarten were likely to 
continue showing patterns of these struggles throughout their elementary years, with the clear implication 
being need for additional assistance prior to the end of kindergarten (p. 319). In Screening for Mathematical 
Difficulties in K-3 Students, Gersten, Clarke, Haymond, and Jordan (2011) illuminate the serious need for 
early math screening and note that specific components of efficient assessments remain unclear, with one 
problem being how can we gain the maximum amount of information in the minimum amount of time (p. 15). 
The authors suggest “any assessment instrument must be guided by findings from developmental and 
cognitive psychology. ..and by mathematics educators’ expertise” (p. 4). 

Native Numbers© is adaptive and mastery-based and in this sense is both a curriculum and an assessment. 
Tasks are designed specifically around the concept of a mental number line on which representations of 
numeric quantities and magnitude are manipulated (Berch, 2005; Dehaene, 2011; Griffin, Case, and Siegler, 
1994). Order, presentation and mastery are based on number sense development (Griffin, Case, and Siegler, 
1994), as well as the cognitive models of Perceptual Control Theory (Powers, 1998), Skill Theory (Schwartz 
& Fisher, 2004) and Flow (Nakamura & Csikszentmihalyi, 2009). Native Numbers’© sequencing of skills, 
with unlimited attempts to progress through proficiency to mastery, fits the model Pellegrino, Chudowsky 
and Glaser (2001) describe as “intelligent tutoring systems (that) have a strong cognitive research base and 
offer opportunities for integrating formative and summative assessments, as well as measuring growth” (p. 
257). Shepard, Daro, and Stancavage (2013) contend that learning progressions are “an advancement beyond 
traditional scope and sequence schema. . .document how learning typically unfolds. . .ensure the close 
connections between assessment and instruction. . .(and) hold promise for the deepening of student learning” 
(p. 137). At face value. Native Numbers© holds promise to meet all the aforementioned current needs. 


3. INSIGHTS FROM ACTION RESEARCH AND PILOT STUDIES 

Goals of my independent study were to: provide the developers feedback from teachers and students on the 
usability of the app and web-based dashboard; and investigate our hypothesis that raw scores taken from the 
dashboard would correlate with scores of identical tasks on a validated math screener. The study included a 
pilot in a first grade classroom and a formal study in three kindergarten classrooms. My district agreed I 
could provide anecdotal informal feedback, but ultimately denied a formal study. The following year, at a 
different location, the second iteration of this study included: formal study of two kindergarten classrooms, a 
pilot study of two first grade classrooms and informal observations of two second grade classrooms. 
Complications encountered during the pilot study postponed the formal study until the next school year; the 
study currently in progress. Insights from all studies to date are combined under the headings of: The 
Possibilities of Adaptive Software with Embedded Assessment; Design and Analyses of Computer Software 
Models; and Collaborative Research Networks Support All Stakeholders. 

3.1 The Possibilities of Adaptive Software with Embedded Assessment 

In each of the studies specific observations held true regardless of whether I was the observer or a fellow 
teacher observed her own students. These observations warrant further study; study, which because of the 
number of observations required, could be difficult to conduct through traditional paper and pencil 
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assessment. Students who had historically indicated difficulty with math skills all made significant progress, 
but sometimes this progress required multitudes of tasks; in one case over three thousand to complete just 
one of the 25 activities. The reality of a classroom setting makes it impossible for any one teacher to provide 
three thousand attempts to show proficiency or fluency in one concept. Pellegrino et al (2001) describe this 
exact scenario: “the most useful kinds of assessment. . .support a process of individualized instruction, allow 
for student interaction, collect rich diagnostic data, and provide timely feedback. . ..(S)ignificant amounts of 
information must be collected, interpreted, and reported. No individual... could realistically be expected to 
handle the information flow, analysis demands, and decision-making burdens involved without technological 
support” (p. 272). Across the three grade levels and schools, the progression of learning was extremely 
varied. No two students learn in the same way over the same period of time. Use of computer adaptive 
software has the ability to make this differentiation feasible. In the five kindergarten classrooms, a total of 92 
students used the app for approximately three weeks for approximately the same amount of time. Of those 
92 students 1 3 students completed the 25 skill sets, reaching the level of fluency; 22 completed 1 3 of the skill 
sets with at least proficiency; and 10 students were able to complete only three or fewer skill sets. 
Combining the number of tasks of the 13 students who complete the app resulted in over 53000 tasks. It 
would be unrealistic to believe a classroom teacher could provide even just 13 students with 53000 drills or 
problems in three weeks just a few minutes per day, much less assess the results. 

Another salient observation is that many of the students had significant differences in the number of tasks 
it took to reach fluency in counting up by one and counting down by one. Comparing just these two skills 
across the classrooms, 15 out of 92 kindergarten students completed these with fluency. The difference 
between the number of tasks required to reach fluency in counting down over counting up ranged from 0 to 
75 with a mean of 36; 36 out of 52 first grade students completed skills with a mean of 32 and a range 
between 0 and 120; and 26 out of 32 second grade students completed both skill sets with a mean of 20 and a 
range from 0 to 125. This data highlights the significant difference in learning progression of students. Of 
particular interest were the students whose previous classroom performance indicated a high level of 
mathematical understanding, yet the number of tasks to reach fluency in counting down was much higher 
than expected; this was particularly true for the subset of students who also had recognized difficulties of 
attention. Scant few studies exist on the cognitive process of subtraction in elementary students and these 
small findings warrant further investigation. (See Barrouillet, Mignon & Thevenot, 2008, for literature 
regarding the cognitive process of subtraction, working memory and discussion of competing theories). 

3.2 Design and Analyses of Computer Software Models 

The pilot study of the first grade classrooms at the second school included a cross over design whereby all 
students would be assessed with a validated screener utilizing pre-, mid-, and post-tests. This design 
eliminates classroom variables such as ongoing instruction. One of the difficulties encountered in the pilot 
study was the initial analyses of data from the validated screener showed most students were at ceiling. 
Furthermore, at the mid-test, little statistical significance was found. Why students were at ceiling on the 
validated screener yet needed multiple tasks to reach proficiency is puzzling and suggests the need for a 
closer look at micro-skill development. The number of variables is great; however one possibility could be 
that the screeners are validated using means of groups. It may be that computer adaptive assessment counters 
floor and ceiling effects of paper and pencil or human administered tests (de Beer, 2010, p. 243). One 
methodology researchers are exploring is using meta-analyses of single subjects; each student is their own 
case and then multiple cases are analyzed. The Journal of School Psychology recently dedicated a special 
issue to this subject (2014). Byiers, Reichle and Symons (2012) offer an explanation of how single subject 
designs can be used for evidence based practice. Forbes, Ross and Chesser (2011) remind us we must keep 
the individual at the heart of each study, “(e)ven if group statistics do not detect individual learning gains, or 
even suggest no gain through lack of statistical significance, individual gains are not insignificant at a 
practical level” (p. 171). What Works Clearing House has issued standards for determining what constitutes 
a single case design as well as standards for visual analyses (Kratochwill. Hitchcock, Horner, Levin, Odom, 
Rindskopf, & Shadish, 2010). Heyvaert and Onghena (2014) explain how to design various intervention 
methods for measuring effectiveness of an intervention and also recommend combining both visual and 
statistical analyses of the data. Additionally, there is a rich amount of literature addressing Bayesian analytic 
methods for single subject designs. For a brief idea of how Bayesian models can inform analyses of single 
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subject designs see: de Vries and Morey (2013), Ferron, Farmer and Owens (2010), and Rindskopf (2014). 
Vos (2007) provides a thorough explanation of how to use Bayesian methods to determine concept learning, 
in particular in determining mastery. Meta-analyses of single subject design studies, analyzed using both 
visual and statistical methods for validating computer adaptive software are warranted, in particular those 
using various Bayes models. This type of analyses will most likely require a collaborative relationship with a 
statistician well versed in Bayesian models. Computer technology that is constructed with cognitive models 
is complex and may very well require complex methods of validation and collaboration. 

3.3 Collaborative Research Networks Benefit All Stakeholders 

One means of incorporating pedagogy in analyses is to include teachers in collaborative research networks. 
Educators do conduct research and the field of action research is rich with data. The research that typically 
occurs in classrooms is considered action research which generally is not designed in the way that 
experimental research is conducted, there may or may not be control groups, and the participant (or sample) 
sizes are not normally large enough to provide statistical power (Sigler, 2009, p.23). The collaborations 
within my research occurred through networks established by the MBE program at UTA; in particular the 
collaborative partnership of the Research Schools Network (See Schwartz & Gerlach, 2011). Many 
collaborations similar to the MBE framework exist and studies from these educator and researcher 
collaborations are emerging: some completed while the educator is enrolled in a graduate program, fewer 
from classroom educators not in one of these programs (Cornelissen, van Swet, Beijaard, & Bergen, 2013). It 
isn’t always the researcher who contacts a school district; I contacted the developers and asked to research 
their app. Regardless of how the collaborations are established, every stakeholder has an equally vested 
interest in researching educational tools; there is no room for hierarchy. Connell presents a model (Figure 
3.1), a “gold standard,” of collaborative networks, the Connell Adaptation Loop. The model is a complex 
adaptive system: each element provides feedback and energy, allowing for growth and emergence. 
Researching educational practice is complex, iterative, and requires time. 
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Figure 1. Connell Adaptation Loop 
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4. CONCLUSION 

In my work to date I have seen what I believe is a model of computer adaptive software with embedded 
assessment that provides students with individualized instruction. What remains is validation of these claims. 
The observations in this paper are from a study in progress and are meant to further dialogue between 
researchers and teachers in an effort to better understand the complexity of designing and researching 
innovative technologies for education. The literature presented here is not exhaustive and the challenges of 
developing and sustaining collaborations has not been included. I believe the fields of neuroscience, the 
learning sciences and digital science offer robust and unique ways to design, research and validate 
innovative, educational technologies by developing ongoing networks of collaborative relationships. I look 
forward to the day when my students no longer say, “It’s just like learning, only fun" but “I learned. It was 
fun!” 
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ABSTRACT 

Colleges provide professional development opportunities to faculty to promote knowledge growth and improvement of 
skills. At the college, Scheduled Development (SD) time for faculty is based on the educational practice and recognition 
of the need for continuous professional development of faculty members. The paper presents a user-centered approach to 
the development of an online environment to enhance and promote active collaboration among faculty in the context of 
professional development. The best practices in software design were adopted in order to reduce the cognitive overload 
on the user so that they can stay engaged and focused on the task. The Scheduled Development (SD) Connect tool is 
under development, implementation and is currently undergoing testing. SDConnect® is a multi-facet targeted software 
tool that aids in dissemination, collaboration, tracking, and integration of faculty SD for post secondary institutions. The 
tool will also enable College faculty and staff to access the information database on SD activities, with the goal that it 
may lead to active collaborations and synergies among the faculty, departments and schools. This project will also enable 
the administration to efficiently administer SD proposals and approvals and to establish historical records and trends of 
the SD activities of faculty for better resource management. SDConnect® is part of an ongoing project called DEAL 
(Diversity in an Environment of Accessible Learning) funded by the College. 
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1. INTRODUCTION 

Scheduled Development (SD) time for faculty is based on the educational practice and recognition of the 
need for continuous professional development of faculty members related to their current or potential roles at 
the college, and in the application of their knowledge, skills and awareness to the development of the college 
as a learning-centered institution. Scheduled development time and activity provides faculty members with 
the necessary opportunities to respond to ever changing learning and development needs of students, 
departments, community groups, the college and the individual faculty members themselves. Faculty 
scheduled development time, allocation and procedures were managed by the various schools and 
departments within guidelines developed and refined by the Joint Committee on Scheduled Development 
(JCSD). The JCSD task is to refine and develop consistent procedures and guidelines for SD planning, 
proposing, reporting, reviewing, dissemination on SD outcomes, distribution of information on SD activities 
and publication of SD project outcomes. 

The College commits a large portion of its annual budget to the activities undertaken by faculty during the 
SD time. However, the outcomes of SD activities are not widely and consistently shared across schools and 
departments. Management of SD proposals and approvals also varies among different schools and 
departments. In this context, SDConnect© was proposed with the objective to develop a framework for 
dissemination, tracking, integration of SD related information to enhance collaboration among faculty in the 
college and help college administration to optimally allocate resources for faculty development. The tool will 
enable Camosun College faculty and staff to access the information database of SD activities, with the goal 
that it may lead to active collaborations and synergies among the faculty, departments and schools. The 
project will enable the SD committee to efficiently administer SD proposals and approvals and to establish 
historical records and trends of the SD activities of faculty. 
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2. BACKGROUND AND MOTIVATION 

The original research questions that have been posed include: “How do we lead and create an online 
environment that promotes collaboration amongst college faculty?’’ and “How do we create a network of 
collaborators?” Collaboration in schools and colleges is multi-faceted by placing value on cooperative work, 
giving latitude to teachers and instructors to influence important aspects of curriculum development, and 
providing release time for collaborative tasks. 

Historically, collaboration among teachers and instructors has neither been taught nor modeled in colleges 
and schools. Also, there has been lack of meaningful support from administration to promote a collaborative 
environment conducive to "learning to teach” or to improve existing skills (Goddard and Tschannen-Moran, 
2007) while there has been evidence of efforts among faculty to increase collaboration (Brownell et al, 1977; 
Louis et al, 1996) in order to improve teaching efficiency (Schachar and Shmuelevitz, 1996), creates a more 
positive attitude towards teaching (Brownell et al, 1977), and induces collegiality and trust among teachers 
(Tschannen-Moran, 2001). Also, there has been evidence that collaboration among faculty improves learning 
outcomes for students as teachers become more engaged and confident on their ability to teach (Hausman and 
Goldring, 2001) while enabling knowledgeable conversation about the process of teaching and learning and 
building collective capacity (Nelson, 2012). 

Fullan (2010) observes that the “collective capacity” built through planned collaboration is a “hidden 
resource” that the educational system has neglected to cultivate. Stoll et al (2006) further asserted that a focus 
on building collective capacity within colleges is key to sustainability in student learning. A school-wide 
culture that expects collaboration among faculty, with an inclusive, genuine and focused practice is essential 
to improve student learning. Therefore, collaboration must be embedded in the routine practice of the 
colleges. 

The literature review by Cordingley et al (2003) asserted that collaborative scheduled professional 
development that emphasizes peer support has positive impact on students, including improved motivation 
and performance. Brownell et al (1977) also affirmed that positive outcomes such as heightened efficacy and 
improved knowledge base are linked to improved student achievements. 

With these issues in mind, a dissemination tool was developed to enhance collaboration by providing an 
environment that informs the college community of Scheduled Development (SD) policies and procedures, 
informs faculty of the opportunities and projects available to them, and also lists the active projects currently 
being undertaken by faculty members. The next step is to implement this capability to allow the Camosun 
community to search through a repository of past and current SD activities thus fostering cooperation and 
collaboration on complementary projects. 

SDConnect© captures the knowledge around tracking and management of SD activities thus allowing the 
JCSD to establish historical trends. The tool will allow faculty to submit a proposal online, time stamp the 
document, record any modification to the status of the SD proposal, and view its status as the SD form moves 
through the system. Finally, the tool will promote liaison between the various departments in the College to 
aid and facilitate some of the administrative duties such as reminders for an ethics application form, request 
for attending conferences and meetings in the pursuit of scholarly activity, travel, etc. 


3. METHODOLOGY 

The methodology adopted was unique since we gathered information by means of extensive consultations and 
meetings to with end users and administrators determine the College culture and existing practices, collect 
documents on processes, procedures and historical information, identify the stakeholders, document business 
needs to identify the needs and expectations, determine project outcomes in order to establish clear and 
achievable objectives, and define assumptions and constraints. 

The assumptions were that if the focus of development is placed on the stakeholders and their needs, it 
will results in a tool that is user-centered and targeted and that provides a framework for dissemination of 
information, tracking of documents through the system for submission and approvals, integration , and 
collaboration among members of the college community. Since the stakeholders are involved in the design, 
they are engaged and motivated, and based on the research evidence, it might lead to active collaborations 
and synergies among the faculty, departments and schools. 
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Once the objectives were identified, armed with the body of knowledge of the literature around the 
importance of collective capacity, and knowledge to build collaboration among teachers and instructors, 
SDConnect© was proposed as a three -phased development: dissemination (phase I), processing (phase II), and 
search (phase III). During each phase of the development, the focus of the study was on targeted stakeholders 
to ensure that the program is focused on their needs as users need to feel in the “flow” for optimal experience 
(Csikszentmihalyi, 1991). 

The usability testing was designed by identifying issues around learnability, and operability of the tool as 
defined by Hadian (2004) to address the accessibility barriers related to the tool usage. Tools seem to enhance 
learning when they are perceived as being invisible. The accessibility guidelines promote the success of the 
tool in terms of learnability, and operability of the tool. This definition was rationalized by Hadian based on 
the ESSI-SCOPE software design guideline. (2014). 

The tool allows faculty to submit a proposal online, time stamp the document, record any modification to 
the status of the SD proposal, and view its status as the SD form moves through the system. Finally, the tool 
promotes liaison between the various departments in the College to aid and facilitate some of the 
administrative duties such as reminders for an ethics application form, request for attending conferences and 
meetings in the pursuit of scholarly activity, travel, etc. This process was crucial to design and implement a 
system that is viable to College needs as well as to the sustainability of the SD Repository Project and its 
scalability for future growth. 

Since collaboration metrics are not easily defined, we have tried a very targeted approach to identity the 
needs of the college. The next step is to implement this capability to allow the Camosun community to search 
through a repository of past and current SD activities thus fostering cooperation and collaboration on 
complementary projects. SDConnect© captures the knowledge around tracking and management of SD 
activities thus allowing the JCSD to establish historical trends. 

A. Need Assessment 

To engage the faculty, we proceeded by delving into a formal inquiry, and handled the project through the 
SOTL (Scholarship of Teaching and Learning) philosophy. This approach enabled us to define the goals and 
objectives of the project based on principles of teaching and learning (Storey, 2009). 

B. Data Collection 

We used a targeted approach through extensive consultation and interviews to determine the College 
culture and existing practices. We collected documents on processes, procedures and historical information; 
identified the stakeholders; documented business needs and expectations; determined project outcomes, 
established clear and achievable objectives; and finally defined assumptions and constraints before beginning 
to plan the implementation of this targeted tool. 

By utilizing the CCAUL (Camosun College Accessibility and Usability Laboratory) for development and 
testing during each cycle, we were able to obtain a perspective on the users and stakeholders (Faculty, Chairs, 
Deans). The OVO Integrated Usability Software was used to create testing environment, questionnaires and 
forms. The faculty can view their current and archived SDs. The process is broken down in to three modules 
[intent, proposal, and report]. Intent is designed to allow faculty to think about when they would like to take 
their SD and how they propose to use the time allocated. The proposal allows them to describe various 
activities the applicant is planning to undertake. Once the instructor finishes his/her SD time, he/she will 
report on the outcome and submit a document outlining the outcome of the activity. These materials are 
uploaded to the server with the intention of sharing with the college community. Following the completion of 
the application from the faculty, the Chair of the Department needs to intervene in the approval process. 
From the Department Chair’s perspective, two separate modules are available. The approval window allows 
the chairs to see if there are any documents waiting to be reviewed. Another interface was designed with the 
idea of allowing Chairs to view the Faculty proposed SD time with the hope of allowing Chairs to look at 
their departmental budget and faculty teaching time. Chairs can filter the result based on the fiscal year. They 
can also identify at a glance whether a document has been approved or is waiting to be approved, or the 
faculty has not submitted their work yet. 

For the Dean’s interface for approving SDs, the Dean gets the notification as an indicator of documents 
waiting for their approval. Another interface was designed with the goal of an overview of the situation in 
each department with a summary of scheduling intent per department. One more interface (not shown) was 
designed to help Deans and their executive assistants to identify documents that have been sitting in the 
system and have not been processed due to various reasons such as lack of a Chair’s intervention/decision. 
They could actually take on the role of the chair and push the document through the system. 
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4. CONCLUDING REMARKS 

SDConnect© has been introduced as a pilot project At Camosun College. The tool has enabled Camosun 
College faculty and staff to access the information database of SD activities, with the goal that it may lead to 
active collaborations and synergies among the faculty, departments and schools. The result of the study has 
enabled us to determine the operational and usability issues before we took the system live. We applied the 
same approach for submission and approval process of SD documents. The feedback from faculty has been 
overwhelmingly positive. Currently, SDConnect© has been in beta testing for three fiscal periods. The 
preliminary results and feedback from faculty and administration has been very positive and further 
improvements and testing are planned to improve the tool and the management of the database. The proposed 
tool is of interest to all colleges and schools that have an active professional and scheduled development 
program and intend to streamline the management of resources and outcomes and promote active 
collaboration among faculty. The practices described in the paper are innovative as the development of the 
tool has followed the best design practices for software development in terms of accessibility and usability. 
The paper is a work-in-progress as we are planning the final phase of development and testing is required to 
properly access the improvements in faculty collaboration and resource management. 
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ABSTRACT 

Gaming activities are an integral part of the human learning process, in particular for children. Game -based learning 
focuses on motivation and children's engagement towards learning. Educational game-based activities are becoming 
effective strategies to enhance the learning process. 

This paper presents an educational activity focusing to merge mobile computing devices and Augmented Reality, as a 
means to engage student in collaborative learning towards the Aesthetical Periods of Music History. The Musical Peddy - 
paper proposes student to use their mobile computing devices (smartphones/ tablets) to find eight stations scattered in a 
set location, and then use their music literacy skills to find the correct answer in each station. Collaborative work is 
required to find the stations and the answers to the clues. 

The musical Peddy-paper was developed mixing Geo-location and Augmented Reality concepts. Geo-location Points of 
Interest (POI’s) where set through Hoppala platform. Augmented Reality browsing and QRCode reading is supported by 
the LAYAR platform. 

In this paper, we introduce the concept of game-based activities as a tool to promote motivation and engagement towards 
learning, and fully explain the development process of the activity. 

We conclude this paper presenting the conclusions contemplating the use of this activity among a group of children from 
the 2 nd Cycle of Basic Education in Basic School Maria Manuela de Sa, Matosinhos - Portugal. 
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1. INTRODUCTION 

Gaming and ludic activities can act as tools to enhance student participation, facilitating cooperative learning. 
These game-based educational activities can be more effective than the traditional teaching methods and are 
an effective approach to promote success (Criado, Salgado et al. 2010). 

Portuguese public schools, at present, have large, non -homogeneous classes. It’s frequent to find non- 
motivated students towards the learning processes, concerning different areas. Traditional teaching methods 
sometimes seem to be an ineffective way to promote engagement and motivation to these students. Game- 
based educational activities supported by mobile computing devices can act as a trigger to added success, 
collaborative learning and improved learning outcomes. According to Jonassen (2007) students learn to 
think significantly, with the thought-activated activities that may be provided by computer or teachers. The 
Musical Peddy-paper intends to act as an extension activity based on innovative non-formal educational 
approach, promoting fun, experiential, playful and collaborative learning. 

This paper introduces the concepts of game-based activities, geographical Points of Interest (POI’s), Geo- 
location and Augmented Reality, mixed and applied in educational context. It fully describes the 
development process of the Musical Peddy-paper activity focusing to an audience of Music Education fifth 
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grade students. We conclude, presenting the preliminary conclusions from the activity conducted among a 
sample group of students at EB Maria Manuela de Sa, Matosinhos. 

The paper is organized as follows: Section 2 introduces the concepts of game-based activities as tools to 
engage and motivate students towards learning. Section 3 describes the Musical Peddy-paper activity 
development with the HOPPALA and LAYAR platforms. Section 4 describes the implementation process in 
an educational environment. Possible learning outcomes and conclusions are presented in section 5. 


2. GAME-BASED ACTIVITIES TO ENHANCE LEARNING 

Games have the power to teach, to train, to educate and to bring people together. In The Republic , Plato drew 
the connection between play and education as he recommended the use of games for the education of 
children. In early learning stages as preschool or kindergarten, games are taught as socializing activities and 
preparation for further learning (Michael and Chen 2006). 

According to Smith (2007) active learning theorists encourage teachers to consider the motivational 
context for students in their classes. The premise is that students learn best when they feel the need to know. 
Activities that provide rewards or prizes act as a motivational force. The actual generation of students is 
characterized as having low-thresholds for boredom and short attention spans. In this context, interaction and 
group activities are becoming essential pedagogical practices. 

These activities are well supported by recent Information and Communication Technologies (ICT), such 
as Augmented Reality and handheld mobile computing devices as smartphones and tablets. Augmented 
Reality is getting more important every day, encouraged by the new smartphones and tablet revolution. 
According to Zarzuela, Pernas et al. (2013) these devices will define a new way of seeing, thinking and 
understanding reality. 

Actual students have known digital technology from early ages, and are commonly identified as the 
“digital native generation” (Vasconcelos and Ribeiro 2013). These students previous knowledge can be 
harnessed to promote learning, blending technology and curricula in different areas. 

This approach is closely related to basic constructivism principles, focusing on activities and 
environments over learning objects itself. According to Wangpipatwong and Papasratorn (2009) quoting 
Parker and Becker (2003) and Tynjala (1999), students achieve better results in constructivist environments. 
In constructivism, knowledge is not external to the individual, so that learning involves the construction of 
personal knowledge from experience and prior knowledge. It appears that the constructivist learning is an 
approach that effectively motivates students, allowing a learning process more active, exploratory and 
interactive. In other words, through the learning process, students construct their own knowledge. 

The Musical Peddy-paper aims to promote a constructivist learning environment based on group work 
methodology by offering a game-based recreational activity. The activity focuses on the role of previous 
student knowledge (both music and ICT); the role of the context and the experience itself; the interactive and 
cooperative character of learning; the new roles assumed by students and teachers and the involvement and 
reflexing of the student on the undertaken task. 

The activity focuses on the use of mobile computing devices. Geolocation and Augmented Reality to 
enhance student engagement and promote learning in the area of music education. 


3. ACTIVITY DEVELOPMENT 

Peddy-paper is known as a pedestrian orientation proof, designed for teams, consisting of a path to a number 
of points or stations attached to questions concerning a subject, in this case. Music History related. The 
Peddy-paper is a playful activity, usually linked to the acquisition of knowledge about a certain topic. The 
Musical Peddy-paper activity was developed with the HOPPALA and LAYAR platforms and designed to 
remix concepts of Geolocation, and Augmented Reality as support to find and interact with eight stations, 
spread within the physical school area. Through AR and the Layar app, students can find the real-world 
geographic location of an object and interact with it. Lor the purpose of the activity, QRCodes were used to 
display web pages containing a music related question and some clues that pointed to the correct answer. 
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3.1 Hoppala Platform 

Hoppala 1 Augmented Reality (AR) Content Platform is open to all major mobile ar browsers, such as Layar, 
Wikitude and Junaio. Hoppala Augmentation provides an easy-to-use graphical web interface to create 
augmented reality contents. This graphical interface allows users to create and configure Points of Interest 
(POI’s) and export them as API URLs configured to serve as “getPOIs response” in the ar browsers. The 
interface includes a dashboard to create and name the overlay URL’s, featuring title, name and the URLs 
Overlay address. Each overlay is programed in a simple interface, by adding and configuring the required 
augments, viewing the geographical location in map or satellite view. The platform allows users to configure 
each augment by adding titles, up to three levels of description, thumbnails, footnotes and filter values. These 
configurations export to the chosen augmented reality browser, in this case, Layar. 

3.2 Layar Platform 

Once all required POI’s are set in Hoppala, it is required to choose one of the three augmented Reality 
Browsers, namely, Layar, Wikitude or Junaio. We have chosen Layar, which indexes the Points of Interest 
with associated information and URLs. These points are picked up by the Layar browser on a mobile device, 
to create the 3D overlay on reality. Layar doesn’t host the files itself, these are hosted by Hoppala that also 
supports the editing of POIs and metadata. The steps involved in creating an augmentation layer with 
Hoppala and Layar are as follows: 

• Create Hoppala account and the required layer; 

• Create a Layar account and create a new Geo Layer; 

• Associate the Layar Geo-layer with the Hoppala layer and add extra information. 

The Layar interface allows configuring and to add information through a set of menus, such as: 

• Overview; 

• API; 

• Metadata; 

• Graphics; 

• Settings; 

• Permissions. 

The Layar web interface also allows edition, testing, consulting errors/ logs and to view usage statistics. 
The overall process is quite straightforward and allows users to create Geo-location layers without 
programming knowledge. Linal steps include the layer publication. 


4. ACTIVITY PREPARATION 

Lor our Peddy-paper activity, eight geographical positions were chosen within the school area, as one of the 
requirements were to confine the area activity in a secure location. Each of these positions would act as a 
station. The first seven stations displayed a printed QRCode with one music question and some clues to find 
the correct answer. The last station prompted the users to an online form, where they write the answers, the 
team member names, class, and other questions concerning the Peddy-paper activity itself. 

Previously to the activity, students received a simple brochure with clear instructions on how to perform 
the activity. The requested material was the brochure itself and a pencil. The principal requirement was that 
one of the team members should have a mobile computing device (smartphone/ tablet) featuring the app 
Layar and capable of GPS and exterior internet access. 

The activity was available in a period of four days and the students had time to organize their teams and 
prepare their quest. 

The activity was proposed in a class of fifth grade students, on EB Maria Manuela de Sa - Matosinhos. 
Live teams were organized and all accomplished successfully the challenge, overcoming the technology and 
musical challenges. 


1 www.hoppala-agency.com 
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5. CONCLUSION 

Musical Peddy-paper activity was held during the last week of the school year, so there was little time to 
implement a thorough case-study analysis and to conduct student interviews. However, relevant data 
elements were collected from direct observation and from the online forms filled by the students. The five 
teams were supervised by teachers while conducting the activity. Direct observation revealed high levels of 
enthusiasm and engagement towards the activity itself as great concern in achieving good time results and 
accurate answers in each station. In each team students shared their knowledge concerning the specific Music 
History questions and some even researched their notebooks or textbooks to find relevant information. Data 
gathered from Google Drive forms filled in the last checkpoint revealed that all the Geo-located stations were 
found and all the required answers concerning the music questions were also accurate. Regarding the mobile 
computing devices used, data show that students used five Android based smarthphones. The Aesthetical 
Periods of Music History are key points to understand music evolution over time and how music integrates 
itself in different cultural contexts. Activity driven learning outcomes resided mostly on the application of the 
musical knowledge acquired along the school year in different contexts. Answer accuracy shows that, in each 
team, students were able to correctly identify twentieth century composers, musical instruments and musical 
styles, through clues involving sound, image, video and graphics. 

These preliminary observations and student feedback, suggest that this kind of game-based activity may 
contribute to added interest and motivation towards learning and scholarship outcomes, by promoting the use 
of technology and previous student knowledge to consolidate specific learning objectives. 

Further work is required to perceive the impact of these activities in student scholar outcomes, focusing 
on the potential benefits on their scientific knowledge, motivation and engagement. 
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ABSTRACT 

This research explored what it is like for university students to participate in a Massive Open Online Course (MOOC) as 
part of their undergraduate course load. We report on some of the temporal dimensions of students’ learning experiences 
as they undertook the MOOC during a regular, campus-based university term. The research is situated in a 
“phenomenology of practice”, a form of qualitative inquiry that eschews participant opinions and instead gathers and 
focuses on his or her lived experience descriptions (LEDs), that is, recollected, everyday moments that transpired for a 
student while learning in the MOOC environment. In the paper we present several of these descriptive snapshots of the 
lived world of the MOOC for undergraduate students and for each LED, we offer a brief phenomenological reflection on 
the theme of temporality. 


KEYWORDS 

Learning experiences, MOOCs, phenomenology, temporality 


1. INTRODUCTION 

Some scholars claim that Massive Open Online Courses (MOOCs) offer unprecedented openness, democratic 
pedagogies, less hierarchical knowledge creation, and unimagined scalability. Others suggest that these large- 
scale courses represent an historic shift in how education will be designed and delivered in the future. 
However, research has yet to confirm or refute the bold claims rationalizing the popularity and efficacy of 
these big virtual learning environments or their disruptive, game-changing potential for education. As Neil 
Selwyn (2009) has argued regarding strident claims made in advance of other educational technology trends, 
a “counterbalance to some of the more hyperbolic elements of [the] discourse” (p. 74) is needed, and in 
particular, through providing accounts of the complex realities of learners’ actual experiences. This research 
offers one such account. 

The study is based in a larger qualitative research project that collected and analyzed students’ everyday 
experiences of learning in a Massive Open Online Course, specifically the first offering of Dino 101: 
Dinosaur Paleobiology (DinolOl), taught in Fall 2013 by Dr. Philip Currie and Ms. Betsy Kruk of the 
University of Alberta (UofA), Canada. DinolOl was Canada’s first MOOC to offer course credit to UofA 
students as well as credit-by-proxy for a fee to participants from across the world. In its Fall 2013 offering, 
Dino 101 attracted more than 23 thousand participants from around the world (University of Alberta 2014), 
of which 450 were campus-based undergraduate students at the UofA. The course was built using the 
Coursera backbone and consisted of 12 weekly lessons with an estimated workload of 3-5 hours/week for 
non-credit participants, and 7-10 hours/week for credit. When the course opened, all 12 lessons were 
released, and although students were encouraged to follow the prescribed timeline, they were free to 
complete the course at their own pace. Each week included short video lectures with integrated quizzes, 
occasional interactives, as well as optional discussion boards and online student Wikis. 

In our study, four major learning modalities were identified and examined separately in order to reveal 
similarities as well as differences between their experiences: 1 ) Students undertaking the course for free; 2) 
Students registered for accreditation for a fee; 3) UofA students enrolled in the MOOC for credit as either a 
fully online course (PALEO 200); and 4) UofA students enrolled in a blended version of DinolOl with a 
weekly in-person seminar (PALEO 201). All together, we conducted thirty in-depth phenomenological 
interviews with students representing each of these four categories. This paper focuses on the experiences of 
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undergraduate students who undertook DinolOl as an online-only course (PALEO 200) during a regular, 
campus-based university term. In particular, we focus on these students’ lived sense of time in and around 
this MOOC. 


2. CONTEXT AND METHODOLOGY 

2.1 Research Context and Questions Addressed 

Empirical studies of students’ MOOC learning experiences are still limited. In the context of Siemens & 
Downes (2011) Connectivism and Connective Knowledge 2011 cMOOC, Littlejohn (2013) conducted a 
mixed-method study about learners’ engagement patterns and identified three levels of engagement: active, 
linkers and passive participants. These results are comparable to Kizilcec, Piech & Schneider’s (2013, April) 
learning analytics findings across three xMOOCs; their study showed four distinct student engagement 
trajectories: completers, auditors, disengaged learners and samplers. In two other xMOOC data mining 
studies of “first MOOCs” (edX’s “Circuits and Electronics” and Duke’s “Bioelectricity”), concern was 
expressed regarding low completion rates, and suggested “persistence” was a key variable determining 
student success (Belanger and Thorton 2013; Breslow et al 2013). Belanger and Thorton (2013) noted that a 
MOOC’s time commitments might prove prohibitive for some students. 

MOOC’s prominently self-paced feature has directed some researchers to attribute students’ persistence 
to the availability of study time and to the discipline of self-direction and regulation. A California-based 
study (Tornsaufer 2013), that compared student retention rate among different social backgrounds in the 
context community college online credit courses, concluded that further study is needed to investigate if lack 
of time is a reason of dropping-out. In their recent review of the MOOC completion rate literature, Khalil and 
Ebner (2014) echoed Tornsuafer’s call for further research in this area. Both quantitative and qualitative 
researchers have recognized the time-sensitive nature of MOOC learning by, for instance, identifying the 
significance of students’ early engagement in online discussion forums (Yang 2014) and noting the time 
needed for a new learner to establish a sense of learning community (Waite et al 2013). A recent case study 
(Chen and Chen 2014) revealed students’ concerns over dropping a MOOC, and suggested that low retention 
rate may be more a matter of time management rather than the shortage of time. Along these lines, some 
researchers have focused efforts on developing learning assistive software to support students in managing 
their time in a MOOC (Gutierrez-Rojas et al 2014; Alario-Hoyos et al 2014). However, no studies to-date 
have methodically addressed the everyday “realities of learners’ actual experiences” (Selwyn 2009, p.74) 
related to the temporal dimensions of a MOOC. 

The authors’ previous study of free MOOC completers examined some of the lived relational dimensions 
of MOOCs and found that an intimate and pedagogically powerful tutorial sphere seemed to develop for 
some students while watching the video lectures (Adams et al 2014). In the current study, given MOOC’s 
low retention rates and its possible relationship to time demands and limitations, we examine the temporal 
dimensions of one MOOC, specifically, how time is experienced and lived for campus-based students taking 
DinolOl as part of their regular full-time program. Our intent is to tease out possible differences between a 
MOOC and a regular university course with reference to time. Thus this research asks: What is it like for 
undergraduate students’ to take a MOOC for credit as part of their campus-based program and how is time 
lived in the unfolding of this experience? 

2.2 Methodology and Data Sources 

In order to study the lived experiences of university students learning in a MOOC, we adopted Max van 
Manen’s (2014) “phenomenology of practice”. This qualitative research approach is based in continental 
philosophy and employs a variety of human science as well as philosophical methods. It aims to explore the 
everyday structures of “pre-reflective” human involvements, i.e., how we experience our everyday lives 
rather than how we may conceptualize, theorize, or even rationalize them afterwards. Through gathering 
examples of lived-through moments or “lived experiences”, phenomenology seeks to reveal overlooked, 
unexpected, as well as taken-for-granted dimensions of a particular human experience, here, the phenomenon 
of learning in a MOOC as part of a university student’s full-time studies in a campus-based program. 
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For this study, phenomenological data was generated via in-depth, one-hour long, phenomenological 
interviews with thirty DinolOl completers, either in-person or via Skype. Ten of the thirty participants were 
UofA students. Interviews were transcribed and culled for Lived Experience Descriptions (LEDs). An LED is 
a specific moment or event that a student recollects that occurred in the context of learning in DinolOl, and 
specifically excludes (edits out) personal opinions that the student may offer retrospectively about the 
moment or event, or about MOOCs in general. Phenomenological analysis was then conducted on each LED 
via the application of multiple heuristics including existential analysis and eidetic reflection. Eidetic 
reflection derives from Phenomenology of Practice’s philosophical tradition, and involves performing 
techniques such as comparing the LED with similar but distinctly different experiences, in order to draw out 
unique meaning aspects of the phenomenon. 

Existential analysis examines each LED across the universal themes of lived relation (relationality), lived 
body (corporeality), lived time (temporality), lived space (spatiality), lived things (materiality), and lived 
technology (mediality). Existentials are universal because they are always already at play in everyone’s 
lifeworld (van Manen 2014). For example, relationality describes our sense of community, of contact, of 
intimacy and closeness to others, whether others are physically present with us or not. Indeed, lived relation 
may even be experienced as ones relation with oneself, or of having no sense of relation at all with others. In 
a previous study of students experiences of a variety of xMOOCs (Adams et al 2014), we examined this 
existential in terms of a student’s relational sense of the instructor; we were surprised to discover that some 
students developed a very strong sense of relation to the instructor in the context of the video lectures. In 
conducting our analyses of the DinolOl LEDs, specifically those described by the UofA student participants, 
“lived time” surfaced as an interesting existential theme with multiple variations. Phenomenologically 
speaking, “lived” time is distinguishable from “clock” time: an hour seems to pass much more quickly when 
I am enjoying a coffee with my friend, than when I am bored or waiting impatiently for someone to arrive. 
While each of these experiences may transpire over the same hour (clock time) and in the same place (a 
cafe), each event is experienced differently in terms of lived time (and indeed, lived space). 

Finally, it is important to note that phenomenology does not aim for generalizability in the usual empirical 
sense. For example, it makes no claim to articulate universal “essences” or immutable themes. Rather, 
phenomenology is concerned with describing possible human experiences and with revealing unique or 
singular meaning aspects that seem to inhere in the phenomenon being studied: “phenomenology describes 
not the factual empirical but the existential empirical meaning structures of a certain phenomenon or event” 
(van Manen 2014, p. 353). Phenomenology attends to what is recognizable and what is singular about a 
particular' human experience. 


3. FINDINGS 

The analysis that follows provides four Lived Experience Descriptions or “snapshots” of the world of 
DinolOl gleaned from our interviews with five PALEO 200 undergraduate students. Each LED is followed 
by a brief phenomenological reflection. The LEDs were selected based on a unique aspect of lived time that 
we uncovered during the existential analysis phase of the project, and in some instances, found across 
multiple LEDs. Other relevant LEDs were not included here due to space constraints. 

3.1 A Four Day Course 

Last term, I was taking five classes at the same time. Since DinolOl was an online course that did not require 
me to go to a class, I wasn ’t always thinking of it. Actually, after registration, I totally forgot about it until 
midterm! All of a sudden, among the craziness of my four other classes, I realized, "Oh no, Ditto ! I forgot 
about Ditto! ” So three days before the midterm, I watched the first half of course content — Chapters 1-6 — all 
at once. Then I reviewed the notes in one day. The fined exam was not much better either: I watched Chapters 
7-12 in one day and spent the next day reviewing right before the exam. Perhaps what I should have done is 
one chapter every single week — but, in reality, I only spent four days in the course. (Paul, DinolOl/PALEO 
200 student ) 
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Although MOOCs usually begin and end on set dates, many allow self-paced learning that affords 
students significant freedom and autonomy. Of course for some students, this can translate into pushing some 
or all of the required learning to the last minute. For Paul, an undergraduate student, DinolOl was one of five 
courses he was taking for credit during the term. His primary interest in taking the course was not the topic of 
Dinosaur Paleobiology per se — although that could certainly be a welcome side effect — but the fact that it 
was fully online. While his other four courses required him to attend weekly classes, DinolOl demanded no 
such routine, and it quickly slipped his mind until midterm time. On the one hand, it seems astonishing that a 
student can consume six weeks of classes — roughly equivalent to 18 instructional hours — in one sitting, 
followed by another day to review, then proceed to a university level midterm; then repeat the same formula 
for the final exam. Of course, one could hardly expect to do this successfully in every subject area! On the 
other hand, perhaps any sense of astonishment one may have at this accomplishment is based on a taken-for- 
granted but unexamined view of the semester system, which typically allocates courses bi- or tri-weekly class 
meeting times spread over three or four months. 

The Latin root for “course” means “a running race or course”, and stems from the verb currere “to run” 
(Online Etymology Dictionary 2014). The word curriculum also derives from this root. A course of learning 
is thus not only about the distance a student has traveled, typically measured by the amount of content or 
“ground” covered, but also the time the journey takes from beginning to end. including its tempo, process, 
motion and rhythm. Clearly a course of learning may not be summarized in a physics equation such as 
“distance equals to speed multiplied by duration/time”, yet as the root currere implies, a modicum of speed is 
nonetheless expected. Indeed, why not allow all students to whiz through their courses at whatever speed 
seems appropriate? In truth, learning — beyond what may be minimally required to pass a multiple choice 
exam — needs both planned and sometimes unexpected moments for a student to slow down, speed up, pause 
and ponder for a while, as well as opportunities for practice and integration over time. Thus while a four day 
course is doable, it hardly seems desirable in the long run. And in the context of Paul’s four-year 
undergraduate program, four days of Dino appeared to be a relatively painless way to achieve credit for one 
of the required science courses for graduation. Here, economies of time seem to supersede educational intent. 

3.2 Learning like a Machine in a Limited Time 

Two weeks before the exam, I started on DinolOl. Each day was quite intensive because of the high volume 
of information. 1 sat at my desk with my earbuds in and a pen and paper in front of me. As I watched the 
video, 1 jotted down what the instructor said. When the video was finished, I closed my laptop, pulled the 
chapter summary from the side, and took out a highlighter to compare and mark what wasn ’t included in the 
videos. Then I made myself a worksheet and put it to the side. Before the next video, I gave myself a break 
before coming back to my worksheet to check what information I still retained. I was learning like a 
machine within such limited time: I just have to keep figuring out the needed information until I feel 
confident. Once there's a topic where I'm confused, I write it on the list and I watch the video again, where I 
listen very carefully to understand what is being scud. If I get side tracked, when I come back to the video so 
much information would have passed. (Kate, DinolOl/PALEO 200 student) 

Computer, headphones and printed notes, paper and pen at hand — Kate deliberately builds a rational 
space and routine for her learning. In doing so, her sense of time also shrinks to the size of the mechanical 
structure built by her. In her words, she is learning “like a machine”. Yet, what does this mean? A machine, 
whether simple or complicated, is programmed to achieve a pre-determined goal; it stays on task unless 
something breaks. As Kate observes, if she gets side tracked, “so much information would have passed”. She 
is highly focused, but only to retain the information for the purposes of the upcoming examination. In this 
sense, learning in DinolOl for her can be compared as an information gathering and checking process, a 
programmable, step-by-step procedure. And yet, perhaps programmed time is less temporal since there is less 
sense of future, only the next repetition of the task. Possible meaningfulness of content withdraws in service 
of efficiency toward another end: passing the exam. Here, the MOOC videos and its chapter summaries 
afford the possibility of a precisely timed and highly efficient consumption of information. But is this the 
kind of learning intended? Or perhaps it is the kind that is encouraged in our current technologized milieu? 
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3.3 Being Slowed Down When Learning With My Brother 

At some point after midterm, I just randomly threw out an idea to my brother, who was also taking DinolOl: 
“Hey, do you think it’d be better if we do this together?’’ And he said, “Sure”. So we started to watch the 
lectures and study together. There was an interactive called “Geological Timescale” with geological time 
periods for us to click and explore. At first I just wanted to rush through and glance it over: It’s impossible 
for them to test on every single detail anyways. But my brother wanted to read through the whole scale: “Do 
you want to read it out loud? ” “OK, sure. ” I clicked and started reading the big terminologies. I was slowed 
down. My brother waited patiently until I finished, then asked if I understood them all. If not, I then took time 
to read them again quietly. Then I handed him the mouse and he started to read. We went through each 
major and sub category. It was a huge project that took us around 40 minutes. ( Michael , DinolOl/PALEO 
200 student) 

Rather than rushing through the interactive exercise on the MOOC, which would have been his habit 
(since it would surely not appear on the test!), Michael was “slowed down” by his brother and his suggestion 
that they read the terminologies out loud. Just as a parent establishes a pedagogical space by reading a 
bedtime story together with their child, the two brother-classmates convened a sphere of learning by reading 
out loud and attending to the strange and somewhat obscure terminologies together. The occasion of this 
shared and memorable learning experience was a unique interactive timescale available in the DinolOl 
MOOC. But too, the timescale and its detail would have been quickly passed over but for the presence of 
another learner with a somewhat different approach to learning. 

Michael observes that, “I was slowed down”. And in slowing himself down to his brother’s speed, a 
“huge project” was accomplished — yet it only took 40 minutes! The student was surprised that he has spent 
so much time on something he would have otherwise overlooked, but too, he recognizes it as a meaningful 
project that was worth the time commitment to see it through. 

3.4 All Checked Off and a Moment of Silence 

It ’s the last lecture for me and my brother. After Betsy thanks us for completing all the video lectures, I stare 
at the blank screen and think, “ That ’s about it. ” All the videos are checked off. Now I don ’t have any more 
video lectures to watch. I feel a bit upset. I could have spent much more time on it. My brother sits still at his 
usual spot on my left. A brief moment of silence between us. “OK, time to review the notes”. We exchange a 
look, stand up, and head back to our own bedrooms to study. The class is not yet done. We have to prepare 
for the on-campus fined exam that takes place in two days. (Michael, DinolOl/PALEO 200 student ) 

In the very last class in a face-to-face course, after a few last remarks about the final exam or paper, the 
instructor wishes students good luck and smiles as she watches them file out the room. The moment may 
quickly dissipate as students approach her about last minutes questions or simply to thank her and chat. In the 
face-to-face context, the end of a course is usually experienced as a fading out — gradually, students prepare 
themselves for its completion. In the case of DinolOl and other video-based MOOCs, the lectures may 
maintain a rather unchanged quality or density throughout the course. Once the final video is watched, the 
MOOC instructor may seem to just suddenly disappear, leaving a blank or frozen image on the screen for the 
student to face. As Michael puts it: “that’s about it.” No more videos to watch. All are checked off. The 
moment triggers a sense of regret: so much more time could have been spent attending to the lectures, 
perhaps exploring the material more deeply. Of course there is nothing stopping him from re-watching the 
videos or pursuing the topic in more depth. Nonetheless the end has been marked with a sudden and 
unexpected decisiveness. 

Each watched lecture has been checked off, like days crossed off on a calendar until the last day arrives. 
Clearly the MOOC space has been designed to afford easy tracking of ones progress and guarantee nothing 
would be left out in the end even for a most careless learner. In some sense, the progress tracking may turn 
MOOC video lecture into a student’s inventory. As Michael reflected, “I don’t have any video lecture to 
watch”. The stock of video lectures is emptied, exhausted, as blank as the screen, not only for him, but also 
for his brother. A moment is silence is shared. Then it is time to move on and return to studying. 
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4. CONCLUSION 

This study presented a preliminary analysis of the existential of lived time as experienced by full-time 
university students while taking a MOOC for credit. We discovered that some students crammed the course 
into a few days, and others methodically plowed through. Here, time seemed to pass in a frantic or machine- 
like blur: the videos and texts were simply information that needed to be quickly and/or efficiently consumed 
for the sole purpose of passing an exam. Yet for others, the experience was also punctuated by surprisingly 
poignant moments — the hour spent working through difficult vocabulary with a brother, the sudden end of 
the MOOC video lessons unambiguously marking the end of an enjoyed course. Here, time slowed down and 
expanded or paused unexpectedly with a regretful air of finality and appreciative reflection. In both cases, the 
moment or event suddenly overflowed with memorable significance. Of course, it is impossible to design for 
such moments — they simply happen. Yet it must also be admitted that new possibilities have been opened by 
MOOCs, providing the unique occasion for such moments of unanticipated significance to unfold. 
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ABSTRACT 

In this paper, we introduce a mathematical model for collaborative learning and the answering process for multiple- 
choice questions. The collaborative learning model is inspired by the Ising spin model and the model for answering 
multiple-choice questions is based on their difficulty level. An intensive simulation study predicts the possibility of 
modification of the Nitta model, which describes the transition of the number of students answering multiple-choice 
questions correctly following discussions among students, using a master equation. 
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1. INTRODUCTION 

In recent years, the theoretical study of teaching-learning processes has attracted the attention of an 
increasing number of scientists. Early studies on those processes have been conducted by psychologists and 
sociologists (Piaget, 1929; Vygotsky, 1978). Although this topic is so complex that there remain many open 
questions, the study of cognitive processes has developed into an active area of multidisciplinary 
investigation as the number of physicists interested in research areas such as economics, social science, 
biology, et cetera, increases. Since Hake (1998) reported that student performance can be enhanced using a 
teaching approach involving collaborative group work, in contrast to traditional non -interactive lectures, the 
processes of learning and understanding physics and mathematics have become the focus of cognitive 
research. In order to further the research in this area, a mathematical model of the teaching-learning process 
is proposed and studied in this paper. 

The aim of using a mathematical model to study the teaching-learning process is to investigate the 
influence of the structure of group work on student achievement and to design effective curricula. We 
classify existing mathematical models of the teaching-learning process into three categories: Ising spin 
modeling. Differential equation modeling, and Stochastic process modeling. 

In relation to mathematical modeling of the teaching-learning process, various models have been 
proposed. For example, Bordogna and Albano (2001) proposed a mathematical model of the teaching- 
learning process in a classroom using the constructive approach. In this model, interactions between students 
and teachers are described with a set of equations similar to those that describe magnetism in materials. 
Pritchard et al. (2008) proposed models of the teaching-learning process using a differential equation. These 
models are based on various theories of learning: tabula rasa, constructivist, and tutoring. They predict an 
improvement in the post-test as a function of the pre-test score, depending on the type of instruction given. 
Although the model itself is quite simple, and an exact solution can be obtained analytically, it fits existing 
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data by sharply determining a parameter. However, these models do not describe collaborative learning 
among students. 

In contrast to the models above that describe long-term learning gain, Nitta’s (2010) model describes a 
short-time learning process. He developed a phenomenological theory of peer instruction and modeled the 
transition of the number of students answering correctly for multiple-choice questions (MCQs) following 
discussions among students, using a master equation. The master equation was simplified analytically and he 
demonstrated that the number of correct answers after peer discussion is approximately given by a simple 
function of the number of correct answers before discussion. The theoretical curve agrees with data obtained 
from lectures implementing the peer instruction. However, it is impossible to ignore the assumption involved 
in the process of simplifying the original master equation. In this paper, we check this reasonability using a 
mathematical learning model, and in particular a model of the answering process for MCQs. 

This paper is organized as follows. First, we briefly review the Nitta model and point out issues related to 
this model. In section 3, we introduce the mathematical learning model used to analyze the Nitta model. In 
the final section, we summarize this paper. 


2. BRIEF REVIEW OF THE NITTA MODEL 

Nitta modeled the transition of the number of students answering correctly for MCQs after discussions 
among students, using the following master equation 

P 2 (c) = Pi (c) + ^ T cd p! (d) - ^ T dc p! (c), (1 ) 

d*c d*c 


where p^c) is the ratio of students who select the correct answer and p 1 (d) is distractors before discussion. 
p 2 (c) is the ratio of students who select the correct answer after discussion. Note that p 12 (c) = 1 — 
Hd*cPl,2 (fO- T cd (0<T cd < 1) is the transition probability of selecting distractors before discussion to 
selecting the correct answer after discussion. With the assumption of T cd — f and T dc — 0, equation (1) can 
be rewritten as 


P 2 (c) = Pi(c) + T(l -PiCc)). (2) 

Furthermore, f is assumed to expand into the power series of p^c) as 

T = K + fciPi(c) + fc 2 (Pi(c)) 2 + •■•+ fe n (p 1 (c)) n ---. (3) 

When PxCc) is small enough, k 0 — 0 is derived because the transition probability from selecting 
distractors to selecting the correct answer after discussions is negligible. Then equation (3) becomes 

f =*■ fciPiCc) (4) 

under the linear assumption. Simple analysis leads to the relation 

P 2 = Pi +Pi(l -Pi) ( 5 ) 

with k t = 1 from equations (2) and (4). Here, we rewrite p 12 (c) as p 12 for simplicity. 

Equation (5) is the main result of the Nitta model but we have questions regarding equation (4). When 
p x (c) is not very small, in other words, in the case of an easy question, equation (3) cannot be approximated 
as equation (4). Therefore, the main result (5) of the Nitta model does not include the effect of question 
difficulty. 

In the next section, we introduce a mathematical model for collaborative learning and answering MCQs 
that includes difficulty information. 
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3. MATHEMATICAL LEARNING MODEL 
3.1 Collaborative Learning Model 

Few mathematical models of collaborative learning have thus far been developed. Among those that have 
been developed are, for example, Bordogna and Albano’s (2001) Ising spin model and related models 
(Yasutake, 2011; Ogawa et al., 2013). Our model is based on the Ogawa model. When students engage in 
group discussion, each student has the cognitive impact of the student-student interaction in a group 
described as 


Q SS (0= ^ /y(t)(l - C/t)) 


Sj (t) + 5/t)l 


( 6 ) 




where Cf s (t) is the cognitive impact on the student i at time t, I t j (t) is the student-student interaction, C/t) 
is the confidence of student i, and5/t)(0 < 5/t) < 1) is the knowledge level of student i at time t. The 
student’s knowledge level is updated according to the cognitive impact level as 


f S^t^j + AS with the probability / Z 

5/t + 1) = -j 5/t) with the probability 1/Z (7) 

y5/t) — AS with the probability /Z 


where Z = + 1 + e^ ( c i S< ~ t '> “) and AS = (1 — 2\S — 1/2 1)/10. 


3.2 Mathematical Model for answering MCQs 

In this paper, MCQs have four answer choices. When a student i has a knowledge level 5/t) at the time of 
answering the MCQ with a difficulty level of D(0 < D < 1), the probability of the student choosing a 
correct answer is 


tanh{2(5/t) — £>)} + 1 
Prob = 


( 8 ) 


Equation (8) means that the probability of a student choosing the correct answer is approximately 1/2 
when there is no difference between knowledge level and difficulty level and this probability becomes 1 (or 
0) when the difficulty level is low (or high). 


4. SIMULATION 

Students answer MCQs according to the probability (8) before discussion (at time t = 0) and collaborative 
learning occurs in each group according to the knowledge dynamics (6) and (7). After discussion (we set 
t— 100 in this paper), students again answer MCQs with a knowledge level of 5/100). Using this 
simulation, the percentage of questions answered correctly before and after discussion can be calculated. We 
set the initial knowledge level randomly between 5/0) = 0.01 and 5/0) = 0.99 and made a simulation 
33,000 times for each difficulty level of MCQ. We then averaged the percentage p 1 and p 2 of a question 
answered correctly before and after discussion and averaged the transition probability f. We can calculate k 1 
and compare the assumption /r 1 = 1 set by Nitta. 

The simulation result is summarized in Table 1. As we can see from Table 1, the coefficient fc 1 , which 
was assumed to be 1, varies according to difficulty level D. 
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Table 1. Relations between D,p 1 ,p 2 ,T, and k 1 . 


D 

Pi 

P2 

T 

fci 

0.0 

0.827 

0.943 

0.674 

0.815 

0.2 

0.718 

0.888 

0.648 

0.903 

0.4 

0.573 

0.814 

0.595 

1.038 

0.6 

0.427 

0.712 

0.515 

1.206 

0.8 

0.286 

0.565 

0.400 

1.399 

1.0 

0.172 

0.389 

0.268 

1.558 


5 . CONCLUSION 

We made a simulation using a mathematical model for collaborative learning and answering MCQs in order 
to improve the validity of the assumption introduced in the Nitta model. From our simulation results, the 
coefficient k 1 , which was assumed to be 1, varies according to difficulty level I). This means that Nitta’s 
formula, equation (5), could be modified by introducing difficulty information for MCQs. The theoretical 
curve equation (5) seemed to agree with data obtained from lectures implementing peer instruction, but 
further investigation with a large sample of empirical data could prove our prediction from a simulation 
study. This is future work. 
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ABSTRACT 

Specifically Second Life (SL) among virtual worlds draws attention of researchers to form collaborative learning 
environments (Sutcliffe&Alrayes, 2012) since it could be used as a rich platform to simulate a real environment 
containing many collaborative learning characteristics and interaction tools within itself. Five Stage Model (FSM) 
developed by Salmon, Nie and Edirisingha (2010) is an application development model supporting educators to design 
collaborative environments in SL to promote learning and group studies of learners. The educators may benefit from 
FSM to test and design collaborative learning tasks in SL to support group studies of learners. Form this point of view, in 
this study; we planned a collaborative learning environment design in SL by using FSM. In this context, the research is 
important to form an example in the literature in the matter of designing collaborative learning environments in SL based 
on FSM. 


KEYWORDS 

Five stage model, collaborative learning environment, Second Life. 


1. INTRODUCTION 

Wang (2009) emphasizes two distinct characteristics of a collaborative learning environment related with 
each other as individual responsibility and favorable dependency. For the effective application of 
collaborative learning techniques, it is vital that each learner takes a role actively in the process for the first 
characteristic, and makes contribution by shares they make to the group study. In addition, contribution of 
each learner to the second characteristic makes up the basis of the success of the group as a whole and of the 
individual success, in other words, contribution of each individual is regarded as the success of group study. 

Specifically Second Life (SL) among virtual worlds draws attention of researchers and Salmon to form 
collaborative learning environments (Sutcliffe&Alrayes, 2012). Because, SL which could be used as a rich 
platform to simulate a real environment contains many collaborative learning characteristics and interaction 
tools within itself. Those who learn in SL can be interacted with others who learn three-dimensional objects 
and environment by means of their avatars. As well as they can establish to groups assign roles to be 
represented in the group with the outfits of avatars, and may use multi communication channels such as 
voiced and written message and avatar mimics. In addition, learners who learn this way can be enabled to 
access environment and learning content through various representations such as audio, written and visual 
media and to take a role actively in SL (Andreas, Tsiatsos, Terzidou&Pomportsis, 2010). Educators can 
develop an SL environment supporting collaborative learning by using these tools, and form discussion 
groups, and organize virtual seminars or conferences ((,'ukurba^i, 2012). For carrying out these activities, 
more than one education methods and techniques such as simultaneous brain storm and taking role can be 
utilized in SL environment (Burgess, Slate, Rojas-LeBouef & LaPairire, 2010). In addition to methods and 
techniques to carry out collaborative learning activities, educators can benefit from an application- 
development model such as Salmon’s five-stage model [FSM] (£ukurba§i, 2012). 
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During the 1990s Salmon designed and tested FSM made of five stages as an application development 
model for learning and teaching by means of online learning. She also designed learning tasks for groups at 
each stage of the model. At the same time, this model being effective for mixed learning environment was 
published in Salmon’s study (2000) as online learning and teaching model for the first time (Figure 1). 
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Figure 1. Five-stage model (Salmon, 2000). 

According to Salmon, interaction and learning level between the learners gradually increases for passing 
to an upper step in the stages of the model. In this context, the model stages can be summarized as follows 
(Salmon, Nie&Edirisingha, 2010: 171): 

(i) . Access and motivation : That is to provide basic preconditions for effective participation of 

individuals. In this stage, system access and system use skills should be earned and learners 
should be encouraged for achieving learning by benefiting from a remote group study. 

(ii) . Online socialization: That is the formation of personal identities of learners in online 

environment, and finding other learners to form communication. In this stage, educators should 
guide to enable connection between learners. 

(iii) . Information exchange : Learners share beneficial information and thoughts related to learning 

content and tasks with other learners and exchange information. In this stage, each learner should 
establish collaboration by means of the support of other learners for the task of each learner. 

(iv) . Knowledge construction: Learners can undertake more complex tasks and form discussions. 

Interactions start to contain more collaboration in this stage. 

(v) . Development: Learners seek means to benefit more from the system, and wish to get assistance 

to achieve their learning tasks, to apply and transfer experiences they lived and the things they 
learned in online environment. 

Salmon (2002) designed learning tasks within the group for each stages of LSM which was developed by 
Salmon (2000). As well as, this model was adopted for training in SL by Salmon and others (2010). The 
model was tested, carrying out three different case studies (Ligure 2) with archeology, digital photography, 
media and communication students. It was found that each stage provides a separate learning opportunity. 
This model has been adopted and adapted to different contexts by online teachers, academics and trainers. 
Therefore we may benefit from LSM tested and developed for designing collaborative learning tasks in SL to 
support group studies of learners. In this study, it was aimed to design an example of collaborative learning 
environment in SL using LSM. In this context, the research is important to form an example in the literature 
in the matter of designing collaborative learning environments in SL based on LSM. 
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Figure 2. Five-stage model of teaching and learning in Second Life (Salmon et al., 2010, p. 181) 


2. METHODOLOGY 

In this study, an example of collaborative learning environment was designed through FSM. The participants 
will complete collaborative learning tasks based on FSM both individually and in groups. These groups will 
be determined by the participants. Each group can determine a design topic such as library, entertainment 
places, and conference areas in order to show that how should be a virtual campus. They can make 
expeditions individually and take screen screenshots in SL for this purpose. The groups can prepare 
individual or collaborative products from their screenshots in SL. The individual products can be exhibited in 
a presentation; the collaborative products can be exhibited in a virtual cube. Finally all products displayed by 
the participants in SL will be evaluated and rewarded. In that way, the participants’ motivation is attempted 
to be improved. 

One-on-one studies will conduct in laboratory with participants for two weeks, and connection will make 
to SL from where they are located during the other weeks. A closed group will be formed by the researcher in 
Lacebook social network to inform the participants, provide communication other than study time and to 
collect data distantly, and it is enabled that participants become members of the group. Thus, the participants 
will keep up with the information by means of the group in Lacebook, and continue to communicate with 
both the researcher and one another for the necessary processes in addition to the application. Moreover, the 
researcher will decide with the participants about the study times of the teams by used Lacebook. Details of 
collaborative environment which we will design based on LSM are Table 1 and as follows. 


Table 1 . Details of the collaborative environment which we will design based on FSM 


FSM stages 

Process 

Environment 

Access and motivation 

Information about SL, installing, membership, first entry, basic avatars 
movements 

Lab. 

Online Socialization 

Socialization methods in SL, creating groups, participating to virtual 
conference 

Lab. 

Information Exchange 

Determining group name, leader, design topic and doing task shared in 
group meetings 

SL 

Knowledge Construction 

Completing individual tasks, creating individual product and brainstorm 
for common product 

SL 

Development 

Creating common product, evaluation and rewarding 

SL 
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2.1 Access and Motivation 

In the first stage, the participants will meet face to face with researchers in the laboratory. The research will 
provide information about the environment and applications on installing SL, becoming free member, entry 
to the environment, interface information, basic avatar movements such as sitting, walking, running and 
flying, the waiting in the right location for reaching the area provided for them. 

2.2 Online Socialization 

In the second stage, the participants will meet face to face with researchers in the laboratory. The research 
will provide information about applications on adding friends, accepting friendship, seeing online avatars 
nearby, teleporting near avatars located in different places, joining groups, personalization of avatars, 
designing and transferring between avatars virtual objects. The participants will be divided into groups. In 
addition to these, the participants will participate to a virtual conference about various isles that could be 
visited in SL by given an educator conducting studies in SL. The conference announcements will be made 
through Facebook group. 

2.3 Information Exchange 

In the third stage, each group will meet in SL as online from the place they are located. The researcher can 
enter these meetings via SL. In these meetings each group will decide on a group name, a group leader and a 
design topic, they will do task sharing and they will help each other within the scope of the topic identified. 
The researcher informs the groups about getting screenshots, loading presentation in SL and their expeditions 
they will make. Also the researcher guides the participants about the scope of the topic they selected and 
group members help one another. Furthermore, the researcher shares documents such as locations, 
announcements of events and seminars in SL by means of the Facebook group. 

2.4 Knowledge Construction 

In the fourth stage, the groups will complete individually their expeditions and share screenshots that they 
took for their individual tasks as a presentation area determined to group in SL and in Facebook group. The 
groups will examine individual products within themselves and brainstorm about which screenshots are to be 
transferred to cube faces and try to design this cube which will be a common product in integrity to reflect 
the topic they determined. For this purpose, the researcher helps the participants when the presentations are 
loaded and informs the participants about cube design in SL and to transfer the screenshots onto the surfaces 
of the cube. 

2.5 Development 

In the fifth stage, the groups will complete group cube as common product in integrity to reflect the topic. 
Following, products exhibited by the participants in SL will be evaluated and rewarded. For this, individual 
products are evaluated by the research; common products are evaluated by educators conducting studies in 
SL. As the reward of the selected group product, we will pay the copyright of it for use in the virtual campus 
architecture. In this way, it is given to the group perception reflecting their experiences in SL to real life. 
Also the learners discover the environment for various opportunities such as education, friendship, getting 
occupation. It is important that the reward is meaningful to the participants. Thus for this reward, the 
participants must be informed about how to use the economy of SL in real life. 
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3. CONCLUSION 

In this study, we planned a collaborative learning environment design in SL by using FSM. It may be 
advantageous to make use of SL for collaborative learning environment, because it supports participants from 
geographically different locations to study whit in groups by using various interaction tools such as 
immediate message, body language and mimic movements by means of avatars. But it is important that the 
collaborative learning environment must be designed in a planned way for this environment to ton into an 
advantage. Therefore, the successful progress of the implementation process can be provided by taking 
advantage of an application development model such as FSM which was tested and developed to support the 
learning within groups in SL. It is essential that the researcher should take place actively in SL in all stages, 
and guide participants. Also in the stages of the model, interaction and the learning level between learners 
gradually increase for passing to an upper step (Salmon et al., 2010). Therefore it will be important to the 
successful completion of each stage. 

In future studies, we are planning to application collaborative learning environment designing in this 
study and examine the effectiveness and limitations of this environment. Similar studies which will be 
implemented with different study groups, themes and methods will create additional sources to literature. 
Also the future studies will revealed out advantages and limitations of FSM on the design of collaborative 
learning environment. 
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ABSTRACT 

How to guarantee graduate attributes has become an urgent challenge amid the increasing progress in scientific and 
technological development and the globalization of economic activity. In order to solve these problems, a system is 
required which can visualize learning outcomes in relation to attainment targets, and store and sample records of the 
learning process. It was on this basis that Niigata University commenced development of the Niigata University Bachelor 
Assessment System (NBAS). This study outlines the NBAS and discusses topics for future research. 
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1. BACKGROUND 

How to guarantee graduate attributes has become an urgent challenge amid the increasing progress in 
scientific and technological development and the globalization of economic activity. 

PDP (Personal Development Planning), recommended in the UK, is used as a reference in evaluating 
graduate attributes (QAA.2001). In PDP, students themselves reflect, and foster the ability to plan their own 
lifelong development, on the basis of transcripts of their results provided by the university and records of the 
learning process known as Personal Development Records (PDR), which they themselves have compiled. On 
the other hand, there have also been reports that PDP requires enormous effort and expense and that learning 
outcomes are hard to grasp. 

In order to solve these problems, a system is required which can visualize learning outcomes in relation to 
attainment targets, and store and sample records of the learning process.lt was on this basis that Niigata 
University commenced development of the Niigata University Bachelor Assessment System (NBAS). Using 
this system, the level of attainment of students' learning outcomes in four educational target domains: 
knowledge and understanding, domain specific skills, generic skills and attitude, are displayed on a radar 
chart and can be visualized. Furthermore, the system also has an e-portfolio function and by using this 
function students can organize and reflect on the record of their learning outcomes and plan their learning. 
The NBAS came into operation in 2013. This study outlines the NBAS and discusses topics for future 
research. 


2. NBAS FUNCTION 

2.1 Learning Outcome Visualization Function 

The method of learning outcome visualization was also explained in Ikuta and Gotoh, 2011. First, the 
educational target domain is taken as domain-specific academic knowledge , domain-specific skills, generic 
skills and attitude(See Figure 1), and attainment targets are set at a low level for these educational targets. 
Next, a contribution ratio towards these attainment targets is set for each course in the program. For example, 
course A might contribute 50 percent to knowledge and understanding, 30 percent to domain -specific skills. 
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10 percent to generic skills and 10 percent to attitude. The learning outcome is the total score obtained by 
multiplying this contribution ratio by the number of credits. Hence, in knowledge and understanding, for a 
student with 80 points in course A, the score is 80 points x 0.5 (50 percent) x 2 credits = 80. The possibility 
of understanding learning outcomes displayed and visualized in this way is examined in Ikuta and Gotoh 
(2014). 

The NBAS has a function which displays role models for comparison when individual students self- 
assess their own learning outcomes. By displaying these role models and their own learning outcomes 
simultaneously, students can be made aware of where they fall short(See Figure 2). 
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Figure 1. An Example of Graduate attributes 
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Figure 2. Visualization of Learning outcome 


2.2 e-portfolio Function 

The NBAS includes an e-portfolio function to enable students who have accumulated many development 
records in the learning process to organize these, making it easier for the students to reuse them or look back 
on which learning outcomes they personally feel are valuable. 

Written reports in word files, excel spreadsheets showing assignments, distributed PDFs, and photographs 
of activities and work can be recorded in the e-portfolio. These can be recorded simply by dragging and 
dropping from individual students' computers. Materials distributed by teachers can also be stored in the e- 
portfolio. 

Files can be organized in folders or by means of tags. They can also be sorted by chronological order or 
by category. The process of differentiating between necessary and unnecessary files in itself is an opportunity 
for students to reflect on which aspects of what they have learned have meaning for them. These files can 
also be quoted on sheets compiled for assessment purposes(See Figure 3). 
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Figure 3. e-portfolio 


2.3 Compilation of Assessment Sheets 


Students are regularly assigned the task of compiling assessment sheets. Assessment sheets consist of three 
elements: 1) visualized learning outcomes, 2) comments quoting files from the e-portfolio and 3) free space 
where students can freely develop the screen. On assessment sheets, students carry out self-assessment by 
quoting files in the e-portfolio relating to the meaning of university education and to aspects of learning 
which had value for them personally. Teachers return comments on the computer and advise students in 
interviews(See Figure 4). 


NBAS ** ?t7$At»*i l I 

I fUf J-fBIB rr.B 



can freely develop the screen 


Figure 4. Assessment sheet 


3. RESEARCH ISSUE 

Research issues which appeared after the NBAS was put into practise are set out below. 

Firstly, the possibility was mentioned of making use of the NBAS for improvements to the curriculum by 
the teaching body. In Ikuta and Gotoh, 2011, the authors suggested that “by revising the curriculum, 
sampling attainment targets, and examining evaluation methods and mark allocations for the program as a 
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whole, individual teachers will be able to examine, in a comprehensive way, programs for fostering human 
resources not only from the point of view of knowledge and understanding, but also from the point of view of 
generic skills and attitude, and this will be useful since it can be widely applied to quality assurance in higher 
education." Since the NBAS was put into practise, the authors are more convinced than ever of this aspect. 
That is because the teaching body will be able to perceive the nature of the program they offer when 
developing a role model. It is actually possible to use performance data about students enrolled at the 
university and to compile data by entering false results. However, in this process, it will be possible to obtain 
information about the balance of the curriculum overall, for example, where learning outcomes are 
insufficient because students have a busy schedule in a particular semester and have not prepared enough 
subjects to achieve their important attainment targets. In future, it is to be hoped that there will be more and 
more cases of curriculum improvement by the teaching body. 

Secondly, students will be able to verify their development by means of self-assessment. Here, both 
synchronic and diachronic considerations are included. As far as synchronic considerations are concerned, 
the NBAS is gradually being extended to cover the whole university, and at present students who follow 
programs where the system is used, and students who do not, are mixed up together. Case reports show that, 
on the basis of learning outcome records, students who use the NBAS reflect more by themselves and have a 
higher awareness of the importance of trying to plan their own lifelong development than students who do 
not. For example, students who use the NBAS are concerned about a lack of orientation in their learning and 
want to be provided with specific knowledge and methods to help them to determine their orientation. 
Students who do not use the NBAS, on the other hand, are not particularly concerned, and have a poor 
awareness of the importance of trying to acquire specific knowledge and methods for determining their 
orientation. 

In connection with diachronic considerations, the point at issue is how do students respond to the same 
question, and how does the quality of their response change. At present, students are asked to respond 
continuously to three questions: the meaning of university education, their current ideas about the future, and 
the understanding and skills they have obtained in comparison to before they entered university. It is likely 
that by graduation, thanks to careful consideration and on the basis of learning outcome records, students 
themselves will be able to reflect on how their response to these questions will change, and to clarify how 
much better they will be at planning their own lifelong development. 

Clearly indicate advantages, limitations and possible applications. 
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ABSTRACT 

In the current study, it was examined whether successful learning is related to using different types of media. We 
compared the comprehension of an economic concept in novices ( N = 82) under three conditions: a Wikipedia article, a 
funny, and a serious YouTube video. The media were presented in English which is a foreign language to most of the 
German speaking sample. The funny video turned out to be the condition related to better success in learning while 
individual variables (e.g. Thinking Styles and metacognitive evaluation) were controlled. Regarding metacognition, it 
was further found out that answering by logical thinking correlated negatively, and that answering by processing the 
presented material correlated positively with the number of correct answers. 


KEYWORDS 

Multimedia learning; differences in learning; foreign language; thinking styles; metacognition; humorous learning 


1. INTRODUCTION 

Since currently the scientific language is English, and most research can be found on the internet, it is 
interesting to examine learning via different types of English speaking internet media. Especially non-native 
speaking students are often unable to cope with selecting an appropriate media. Therefore, this study 
compares the comprehension of an unknown concept in English language in three common media on the 
internet. 

1.1 Medial Learning 

According to the “General Factor Model”, there is no difference between a read and a heard foreign -language 
text in terms of comprehension (Leucht et al. 2010). Research on instructional designs (e.g., Sweller 1994; 
Paas et al. 2003) indicates that pictures support a written or spoken text if they are presented in an appropriate 
manner. This means that the individual does not have to switch between two sensual modalities but sees a 
picture and hears a text at the same time, as it is the case in videos (Low & Sweller 2005). Therefore, it is 
expected that understanding an unknown concept is easier with a video than with a text. A lot of videos on 
the internet explain difficult concepts in an easy-going or funny way. Fun and laughing decrease anxiety in 
learners and thus enhance their performance (Stambor 2006). Hence, learning with a funny video might be 
easier than with a serious one. 

1.2 Influences on Learning: Thinking Styles and Metacognition 

Thinking Styles interact with personality, educational specialization, career, current job roles, and 
adaptability (Kolb 2005). Originally, the inventory was developed as a tool for educational and occupational 
consulting (Kolb 1981). The author found the following four styles on two axes: Assimilation vs. 
Accommodation and Divergence vs. Convergence. Recent research about Thinking Styles in combination 
with online learning shows that the style of Assimilation profits the most from reading articles, using 
Wikipedia, and watching videos (Arp et al. 2008). This style combines learning by “Reflecting Observations” 
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(RO) with “Abstraction and Conceptualization” (AC). Those ways of learning are both conceptualized as 
understanding or transformation of knowledge (Atherton 2013). Accommodation oriented persons are rather 
active and pragmatic which means that they use the strategies of “Concrete Experiences” (CE) and “Active 
Experimentation” (AE). Divergence oriented persons are active, but also reflective, and profit from CE and 
RO. Convergence oriented persons use AE and AC. They are pragmatic as well as theoretical oriented which 
enables them to high-quality problem solving (Kolb 1981 ). Taking into account these differences in Thinking 
Styles, they are tested for an influence on learning success. 

It is important for students to evaluate, if they really profit from the medium they use because they have 
to choose it themselves in real life (Antonietti et al. 2014). The skill of thinking about the own cognition is 
called metacognition (Costelloe et al. 2006). Antonietti and colleagues (2014) discussed that learning is 
influenced by metacognitive expertise which can strengthen the effects of media. Thus, it is assumed that the 
observed type of answering (e.g., guessing or answering by presented material) could have an influence on 
the relationship between presented medium and correct answers. 


2. EXPERIMENT 


2.1 Participants 

The experiment included 46 (56.1 %) women and 36 (43.9 %) men between 20 and 46 years. In average the 
sample was 23.90 (SD = 3.60) years old and studied for 6.25 (SI) = 3.11) semesters. At the time of the 
experiment, those 82 people were students at the Leopold-Franzens University Innsbruck. In order to ensure 
a certain level in English, the study only included Psychology students (at Austrian universities, applicants 
for Psychology have to master an obligatory English test) and students of a B2+ English class. The 
participants of the B2+ class studied different non-economic disciplines. In order to detect thematic experts, 
it was also asked for the Psychology students' former and second disciplines. 63 (76.8 %) persons studied 
only Psychology and 9 (11.0 %) persons studied a second subject, too. Persons who did not finish all items or 
stated more than one time that they knew the correct answer from somewhere else (e.g., former studies) were 
dropped out. The sample was invited personally as well as using Web-Mail and Facebook, and oral consent 
was given. All probands had normal or corrected-to-normal vision. 

2.2 Materials and Procedure 

The sample was randomly assigned to one of the different media conditions described below. The 
questionnaire was answered in the same order as shown in the following. The respective medium was 
presented two times with a break of 130 seconds between the first and the second presentation. After a 
second break, the participants were asked to fill in the questionnaire. These breaks should enable the 
participants to process the knowledge gained from the material. This length was chosen because according to 

Preim (1999) 130 seconds are sufficient for cognitive processing. 1 The gathered data were computed using 
SPSS 21. 

2.2.1 Media 

All used media were presented in English language and about the topic of Opportunity Costs. The reading 
passage was taken out of Wikipedia and the videos out of YouTube. Every medium was presented the way it 
was found in the internet. The idea behind using this raw material from those pages was that most students 
use these sources (Kleimann et al. 2008) in order to prepare for tests or the like. Because it was the aim of the 
study to show a real life situation, differences between the three media conditions relating to the written or 
spoken text were part of the approach. It is in their nature that the spoken text of a funny video cannot be the 
same as the spoken text of a serious one because the linguistic style changes with the underlying intention. 
The same is true for a written and a spoken text in comparison. The topic was chosen because it is largely 


Media and questionnaire are available upon request. 
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unknown among Psychology students. Therefore, it was possible to minimalize that the participants apply 
their already existing knowledge to answer the questions. The length of the videos was approximately the 
same, and the time for reading the text was limited to the same length. The videos were presented in full 
screen mode on a 2x3 meter wall with a beamer. The loudness was regulated accordingly to the participants' 
wishes. 

2.2.2 Questionnaire: Description 

German was chosen as language for the multiple choice test because the study should demonstrate a “real- 
life” situation of a student preparing for a German speaking test using English speaking online sources. The 
questionnaire included a short description of the process at the beginning. In the text condition, the text was 
on the second page. The following pages were the same for all experimental conditions. 

2.2.3 Questionnaire: Multiple Choice and Metacognitive Evaluation 

The participants had to choose the correct one out of four possible answers. Three of the six questions were 
rather abstract questions which required a transfer to an example that was not directly part of the presented 
material. The sheet was horizontally presented, and one third of it was folded backwards. Thereby, people 
were not able to see the second column next to the multiple choice questions. In this column, the following 
statements regarding metacognitive evaluation were to be chosen: “it seemed logical”, “I have already heard 
about it somewhere else”, “I knew it from the text/video”, or “I guessed”. The multiple choice questions had 
to be answered before the metacognitive self-report and were similar for all three conditions. 

2.2.4 Questionnaire: Demographical Data and Personal Information 

Afterwards, demographical data were raised. Furthermore, it was asked if the participants studied only 
Psychology or an additionally discipline. Moreover, it was asked if they had studied another discipline before 
they started Psychology. If they were or had been enrolled to an additional one, they had the opportunity to 
fill in which discipline they studied besides, respectively, before Psychology. Due to the information about 
the participants' academically background, it was possible to exclude those who had an extremely higher pre- 
knowledge by attending an economic subject. 

2.2.5 Questionnaire: Thinking Styles 

The last page was an adapted German speaking self-report measurement (Wehner 2014) based on the 
Learning Style Inventory (LSI) by Kolb (1981). On the top of the page, a sentence said: 'If you think about 
learning of new English terms, how was your approach, respectively, attitude one year ago?'. Then, four 
columns and nine rows followed. Each cell contained an adjective. The examiner instructed the participants 
to restart thinking for each row in reference to the sentence above. Then, the participants were asked to assign 
a number between one and four to every term per row (4 = high agreement, 1 = low agreement). Using the 
method of Wehner (2014) the four styles defined by Kolb ( 1981 ) were generated. 

2.3 Results 

In order to find out differences between the different media conditions, while controlling for Thinking Styles 
and metacognition, an One-Way ANCOVA was computed. Results indicated existing differences between the 
different groups ( F (2, 72) = 5.13, p = .01). In the humorous video condition significantly more items were 
answered correctly ( M = 4.89, SD = 0.80) than in the serious video condition (M = 4.31, SI) = 1.07) and the 
written text condition ( M = 4.15, SI) = 0.92). Neither Thinking Styles (which were converted into dummy 
variables) nor metacognitive evaluation had an influence on the relationship between the presented medium 
and the learning outcome. Regarding metacognition, it was additionally examined if the overall number of 
correct answers was related to the other answering types. Interestingly, the number of questions which were 
answered by logical thinking were negatively related to the number of correct answers (r = -.215, p = .03). A 
positive correlation was found between correct given answers and the number of items which were answered 
by learning through material (r = .231, p = .02). 
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3. CONCLUSION 

The positive results of the funny video indicate that humor facilitates learning which substitutes the findings 
of Stambor (2006). Because this seems to be true even for foreign languages and more abstract tasks, funny 
videos should be used for the comprehension of new terms in a foreign language. Future research should 
examine, if this is also true for two different text versions (e.g., funny vs. serious). Likewise, it could be 
compared, if a funny text and a funny video differ from each other. If that was not the case, it would be 
accordant to the actual results of comparing both serious conditions which did not differ here. It would also 
be a proof for the General Factor Model (Leucht et al., 2010). Because the positive effect of the humor 
condition is independent of Thinking Styles and metacognition, the presentation (e.g., funny vs. serious) of 
the foreign-language medium seems to play the bigger role. Another suggestion is that the whole process of 
watching or reading, translating, and answering is so complex that every Thinking Style can profit of its very 
own special abilities. 

Metacognitive self-reports referred to how the multiple choice items were answered. Regarding 
differences in the presented media, no influence of metacognition was detected. However, considering the big 
picture, answering by logical thinking impairs the sum of correctly given answers whereas answering by 
internalized material enhances it. The finding about the logical answering suggests that it is not always an 
expedient solution to answer by logical conclusion. One's own ability of straight reasoning might be 
overestimated. This can be seen as an example of the Dunning -Kruger-Effect which argues that someone who 
is incompetent in a certain social or intellectual field overestimates his or her competences in it. The authors 
did investigations on it in humor, grammar, and logical reasoning. Incompetence in a certain domain leads 
not only to bad performance in it but also to a misperception of the same (Kruger & Dunning 1999). 

A possible point of criticism could be that the three media lack comparability. This is due to a slightly 
different approach to the topic, since the study refers to an everyday-life situation. As already mentioned, the 
choice of language changes with the underlying intention. Therefore, this variability can be seen as part of 
the approach. 

To put it in a nutshell, funny videos, as well as thinking about the material during answering increases the 
chance of good results in the next test independent of one's own Thinking Style. 
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ABSTRACT 

There has been growing interest in the many uses of microblogging in higher education, from sharing research ideas and 
results to teaching and learning. Yet few studies have been undertaken to examine systematically how microblogging 
technologies are used by teachers in a classroom environment. The published studies indicate microblogging tools aide 
collaboration in the classroom. These studies have not addressed in detail how the use of the technology shapes teaching 
practice. The research presented in this paper is an exploratory study that examines how and why instructors use 
microblogging in their classrooms. This study builds upon previous study that looked at how students’ behave when 
using micro-blogging tools. Here, we investigate the lessons learned by instructors who have used this technology as part 
of their resources to connect with students. 


KEYWORDS 

Microblogging, Online Teaching, Learning, User Engagement, Collaboration 


1. INTRODUCTION 

There has been growing interest in the many uses of microblogging in higher education, from sharing 
research ideas and results to teaching and learning. Yet few studies have been undertaken to examine 
systematically how microblogging technologies are used by teachers in the classroom. Most of the studies 
that have been published focus on practical and social applications, usually on Twitter, to enhance pedagogy 
and support collaboration in the classroom. These studies have not addressed in detail how the use of the 
technology shapes teaching practice. The research presented in this paper is an exploratory study that 
examines how and why instructors use microblogging in their classrooms. Seeking to increase student 
engagement and learning, some instructors at Camosun College as well as in other colleges and universities 
are asking students to use Twitter, Facebook, the discussion forum in D2L, or other micro-blogging tools to 
post online questions and comments relevant to course material. Previously, we have investigated and 
published our findings on the students’ behavior when using micro-blogging tools [Froese et al, 2011]. Flere, 
we build upon these findings and investigate the lessons learned by instructors who have used this technology 
as part of their resources to connect with students. We discovered that popularity can be deceiving and tools 
such as Twitter, Facebook and Google+ are not always beneficial to students’ motivation and learning. 
Moreover, if these tools are not used with caution they might hinder the students learning experience. 

Microblogging is an online tool that allows a small amount of text based content to be published in the 
user’s profile page [Java, Song, Finin and Tsend, 2007]. The most popular microblogging tool is Twitter, 
which was launched in 2006 and it allows users to post short messages and update status and useful web 
resources. Microblogging is limited to 140 characters per post on Twitter. The lightweight nature resulting 
from the posting constraint of microblogging makes it possible for people to post quickly and post often. 
Some microblogging tools also allow for multimedia content, where users can exchange small elements of 
multimedia content such as text messages, individual images, and video links [Grosseck and Holotescu 
2008], There is a plethora of studies reported in the literature on the benefits of using microblogging for 
social and pedagogical purposes. Junco, Heiberger, and Loken [20111 have reported that the use of Twitter 
encouraged online participation from some students who otherwise might not have participated in class. 
Similarly, Rankin [2009] noted that the integration of Twitter, as a communication tool, allowed more 
students to participate in classroom discussion than before. Zhao et al. [2009] also suggest that 
micro-blogging increases students’ motivation, engagement and grades. Ebner and Maurer [2009] stated that 
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microblogging allowed learners to stay connected and share information in the eLearn community. 
Microblogging tools were also found to benefit informal learning beyond the classroom. Twitter assignments 
have been demonstrated to promote active learning by helping students relate the course material to their own 
experiences both in and outside the classroom [Junco. Heiberger, & Loken, 2011]. 

Thus, in educational settings, we accept that micro-blogging is used to ask questions, to give opinions, to 
share resources, to change ideas, to reflect, and to foster informal learning outside the classroom [Ebner et al. 
2010], We noticed that in most of current studies, micro-blogging was a mandatory activity for students, 
rewarded with marks, and controlled by their instructors. Despite the promising technological features and 
pedagogical features of microblogging, there is a lack of consensus on the integration of such activities and 
their impact on facilitating learner interaction. Educators have recognized some drawbacks in the use of 
microblogging, such as the potential for it being distractive and addictive [Gossec and Holetescu, 2008]. 

In our previous research [Froese et al, 2011], we analyzed the posts in the chat-room in Connex of a large 
(160 students), first year class in computer science. These students were encouraged but not rewarded for 
micro-blogging, and it was an alternative communication media to email or face-to-face interaction. We 
found out that students communicate through micro-blogging in a quick and concrete fashion, a post is 
usually a short message of 78 characters in average (note that some micro-blogging tools such as Twitter 
limit the size of a post to 140 characters). 91% of the students were casual users with less than 15 posts per 
term whereas only 9% of the students were active micro-bloggers posting between 15 and 40 posts per term. 
We found that that micro-blogging favors peer support. Students were eager to respond to questions if they 
knew the answer, even in cases were a question had already been answered in older topics they would still be 
willing to share answers. We noticed that students were not shy about asking questions from the simple to the 
more complex. Almost 40% of the posts were in the cognitive domain, asking or providing application, 
comprehension, or knowledge information. The remaining 60% of the posts were for social interaction or for 
administrative information related to the class, micro-blogging turned out to be a convenient way of 
socializing and communicating. We noticed that some students were asking more cognitively complex 
questions in the first weeks of the course, contrary to what we would have predicted. One can infer that as 
students become more skillful and familiar with the class material they need less assistance on how to 
interpret concepts and on how to solve problems. However, students consistently asked for help on finding 
information such as the syntax of programming instructions or on finding references to complete 
assignments. However, how to design classroom activities to optimize the positive effects of microblogging 
remains a challenge for educators. The purpose of this study is to examine student learning under a set of 
structured microblogging-based activities from the instructor’s perspective and to identify the affordances 
and constraints of the technology. 


2. METHODOLOGY 

We conducted five interviews with instructors who had used micro-blogging as communication media with 
students. Our participants had experience with tools such as Tweeter; Facebook; Google groups; and 
discussion frameworks such as WebCT, Moodle and D2L. Instructors of face-to-face courses used micro- 
blogging as an alternative communication media, whereas for instructors of online or hybrid courses, it was 
the main communication media. We asked instructors open questions aimed at finding what are the factors 
that lead to successful online micro-blogging discussions. Examples of questions are: tell us about your 
experience with micro-blogging, how do to you keep students engaged in the micro-blogging activity, and 
describe your best/worst experience with micro-blogging. Two researchers lead the interviews, recorded, and 
took notes during the interviews. Interviews were transcribed, coded, and analyzed to discover trends and 
emerging themes. There were two main themes that emerged in the codes of the interviews: instructors’ 
motivation for using micro-blogging and advantages of using micro-blogging. 

Instructors’ motivation: The instructors’ motivation to use micro-blogging tools were to get quicker 
feedback from students, to increase communication with (and among) students outside the classroom, to have 
a voice to the less vocal students, to have a space were students could be more reflective on their input and 
comments, to have a channel for discussions regarding class content or administrative issues, and to create a 
sense of community on an online course. 
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Advantages of using micro-blogging: The use of microblogging leads the student to become active 
learners and to learn effectively without being watched by the instructor, it gives voice to the quiet students 
who otherwise would not express their comment of ideas in class. It provides the student with the opportunity 
of having other student to critic and validates their ideas and with criticism comes a valuable opportunity to 
debate different ideas. This interaction provides an opportunity for weaker students to see others’ work and 
motivates them to aim at higher standard for their own work. 


3. FINDINGS AND RECOMMENDATIONS 

Our findings corroborate with the theories of online teaching and learning when using microblogging tools in 
a classroom setting. For example, according to Downes’ [2006] and Siemens’ [2004] theory of 
connectivisim, learners should be given the opportunity to explore learning and not be overloaded with 
information. The study also confirmed observations that instructors have become facilitators of construction 
and discovery of knowledge by promoting organized exchange and discussions. 

The study further confirmed that instruction needs to be concise and clear and this was crucial to the 
success of incorporating microblogging for online discussions in the course content. Another aspect of 
interest was that the choice of microblogging tool is very important as the familiarity of the tool facilitates 
learning. The study showed that the content used for discussions and their relevancy and challenge were 
important to the participation and engagement of the student. This supports Keller’s proposed ARCS model 
(Attention, Relevance, Confidence, Satisfaction) for motivating learners during learning [Keller, 1983; Keller 
& Suzuki, 1988]. 

The findings for effective use of microblogging were classified into four sub-themes: organization of the 
discussions, definition of clear instructions, the change in the role of the instructor, and how to choose a 
microblogging tool. These subthemes are described as: Organization of the discussions, Define clear 
instructions, the Role of the instructor, Choosing the micro-blogging tool 

• Organization of the discussions - Micro-blogging discussions should be organized by topics so that 
students do not get overwhelmed, students should not be allowed to create new discussion topics, but they 
should have a space to post general questions or requests. Micro-blogging discussions work better with small 
groups, if the discussion is about class content then a group of five or six students will be ideal. For general 
administrative questions larger groups work well. Students seem to enjoy taking turns to moderate the 
discussions in a term. The discussion topics should be judiciously chosen, prepared, and set up well in 
advance, preferably before the course starts, many students nowadays have full time jobs and a family to 
raise and they make an analysis of the cost-benefit of a task before deciding how much time and attention 
they will devote to it. 

• Define clear instructions - Instructors should describe exactly what the students are suppose to do, 
what the policies are, and how the students will be marked. Instructors should indicate exactly how many 
posts students need for participation marks, what is the time frame for posting, and whether there is a reward 
for early posting. Instructors should give examples of what they consider good and bad posts for their course. 
It is strongly advisable to have a colleague to edit the instructions so there is no ambiguity. Each assigned 
discussion should be relevant to the course content and have an objective that is clear for the students. The 
instructor should define policies for the expected online behaviour. Instructors have observed that students 
perform better when they are rewarded by their online participation just like in-classroom participation and 
that marks are the students preferred reward; another attractive reward is to tie the discussion to the questions 
in the exams. 

• The role of the instructor - The role of the instructor in a micro-blogging discussion is more as a 
facilitator and less as a lecturer and should introduce the idea of micro-blogging with excitement, students 
appreciate an enthusiastic instructor and will increase their changes of having a positive learning experience. 
It has been observed that students perform better when instructors maintain an online presence, it seems that 
students get discouraged when the instructor “is never there”. Instructors should synthetize and talk about the 
posts commenting on important issues that might have been rose online. Instructors should be more active 
answering questions at the beginning of the term and then become less dominant of the discussion as time 
progresses to give students the opportunity to create a community of learners where students can support 
each other. The instructor should encourage and acknowledge students to answer questions posted by other 
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students. It is advisable that the instructor reminds the students about their micro-blogging discussion 
commitments, a short email, a comment in class, or a small video would keep the student aware of their to- 
do’s and timelines. 

• Choosing the micro-blogging tool - Instructors should know their target audience. If students are 
already familiar with D2L, Moodle, or Connex the instructor should consider using the discussion forum in 
those tools, it is easier for students to go to “one” place. The instructor should be familiar with the chosen 
tool well in advance so technical issues do not come up by surprise. Students who already know how to use 
the tool have a higher probability of having a positive experience with micro-blogging. Instructors should not 
assume that all students are computer-literate. If possible, in a face-to-face class, the instructor should demo a 
micro-blogging discussion. CONCLUSION and Recommendations 

To summarize, our findings corroborate with the theories of online teaching and learning when using 
microblogging tools in a classroom setting. Learners should be given the opportunity to explore learning and 
not be overloaded with information. Instructors have become facilitators of construction and discovery of 
knowledge by promoting organized exchange and discussions. Content needs to be concise and clear and this 
was crucial to the success of incorporating microblogging for online discussions in the course content. The 
study showed that the content used for discussions and their relevancy and challenge were important to the 
participation and engagement of the student. 
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ABSTRACT 

Recently organizations have begun to realize the potential value in the huge amounts of raw, constantly fluctuating data 
sets that they generate and, with the help of advances in storage and processing technologies, collect. This leads to the 
phenomenon of big data. This data may be stored in structured format in relational database systems, but may also be 
stored in an unstructured format. The analysis of these data sets for the discovery of meaningful patterns which can be 
used to make decisions is known as analytics. Analytics has been enthusiastically adopted by many colleges and 
universities as a tool to improve student success (by identifying situations which call for early intervention), more 
effectively target student recruitment efforts, best allocate institutional resources, etc. This application of analytics in 
higher education is often referred to as learning analytics. While students of post-secondary institutions benefit from 
many of these efforts, their interests do no coincide perfectly with those of the universities and colleges. In this paper we 
suggest that post-secondary students might benefit from the use of analytics which are not controlled by the institutions 
of higher learning - what we call DIY (Do It Yourself) analytics - a set of tools developed specifically to meet the needs 
and preferences of postsecondary students. The research presented in this paper is work in progress. 


KEYWORDS 

Academic analytics, student-centered learning, postsecondary education, learning analytics. 


1. INTRODUCTION 

Recently organizations have begun to realize the potential value in the huge amounts of raw, constantly 
fluctuating operational data sets that they generate and, with the help of advances in storage and processing 
technologies, collect in transactional systems [1], The latest techniques from computer science, mathematics 
and statistics are needed to perform this analysis and generate strategic insights, e.g. about visitors to a 
company’s website for better marketing efforts, resulting in the growing importance of the field of business 
analytics. Both structured data (stored in relational and non-relational database systems) and non-structured 
data can be analyzed using data mining techniques and the results presented using information visualization 
methods to best guide organizations’ decision makers [2, 3, 4], Data mining [12] is sometimes differentiated 
from analytics by the way that analytics tests for specific hypotheses while data mining lacks a hypothesis, 
instead searching large data sets for interesting patterns. Some other experts consider data mining to be a part 
of analytics. 

Analytics has been enthusiastically adopted by many colleges and universities as a tool to improve 
student success (by identifying situations which call for early intervention), more effectively target student 
recruitment efforts, best allocate institutional resources, etc. [5, 6], The evolution of big data and its 
widespread adoption in American higher education has been documented by Picciano [8], Sources of the data 
which can be analyzed include institutional data about students, courses, applicants, however, a particularly 
rich field to mine for data is associated with online courses and Course Management Systems (CMS) [11]. 
Information extracted from a CMS can be quickly assessed for early warning signs of student failure, leading 
to prompt intervention and increased chances of student success as well as higher student engagement. Such 
data can also be used for student assessment and course redesign. 

Academic analytics uses a combination of institutional data, statistical analysis, and predictive modeling 
to create insight which students, instructors, or administrators can use to develop a strategic plan for 
enhancing academic outcomes. The University System of Georgia carried out an experiment using analytic 
techniques to develop an algorithm to predict student completion and withdrawal rates in an online 
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environment. The results helped to confirm that it was possible to predict accurately the likelihood that a 
student would successfully complete an online course [6]. 

Goldstein [9] proposes the term “academic analytics” as an alternative to “business intelligence” in the 
academic realm. He surveys seven areas that analytics can be used in academia: advancement/fundraising; 
business and financing; budget and planning; institutional research; human resources; research 
administration; academic affairs. The Signals project at Purdue University has delivered early successes in 
academic analytics, prompting additional projects and new strategies [10]. A visual analytic tool being used 
for student enrollment is shown in figure 1. Clearly, most of these areas are not of interest to students in post- 
secondary education, except perhaps tangentially. Our DIY approach will concentrate on areas that are 
directly of interest to students. 
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Figure 1. SAS Visual Analytics for student enrollment, (http://www.sas.com/software/visual-analytics/demos/student- 

enrollment.html) 


2. DIY ANALYTICS 

While institutions of higher learning have increasingly relied on learning analytics, and while the use of 
analytics by the institutions can be of help to students (for example, by identifying if they are at risk of failure 
in a course or in a program of study, and providing intervention to help them), the needs and requirements of 
institutions and students are not identical. As an obvious example, students have an interest in enrolling in the 
institution which gives them the best chance to succeed in their chosen field, however a particular institution 
has an interest in getting that student to enroll, even if there is some other university which would better meet 
the student’s needs. 

Consider also the following examples of divergent interests for institutions and the students enrolled in 
the institution. Students would like (all else being equal) to enroll in classes taught by professors that give 
them the best chance of achieving their goals. Institutions (colleges, departments) don’t have any interest in 
steering students towards particular instructors and away from others. On the contrary, the department’s 
interests are best served by having level enrollments in all sections of courses, rather than having some very 
large sections and other very small ones. Another example is in the choice of a field of study within an 
institution. The student’s needs to discover the best program for him or her might not coincide exactly with 
those of the institution, which might want to steer students towards favored programs or away from others 
which might be getting too large. 
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In all of these cases, we propose the introduction of what we call DIY (Do It Yourself) Analytics for 
students. The name refers to the fact that the student himself will be using the analytics tools to make his best 
choices, rather than relying on the institutional filter (the DIY name is not meant to suggest that students will 
be creating/programming these tools themselves, only that they will be the end users). Student -centric 
learning analytics tools should be developed to allow students to reach their academic potential. We have 
identified in the above scenarios several types of insights that students would find valuable in the course of 
their academic careers, but these just scratch the surface. There are many more possibilities that can and 
should be explored. We will be expanding the list of possible topics in future research. 

One issue that arises immediately is - where will the information that will be the input to the analytics 
process come from? In organizations, this is not a problem, since the organizations own the data that they 
generate. The situation is different in this case however, since the students do not generate or own the data 
that they need for DIY Analytics to work. The institutions of higher education could make this information 
publicly available so that it could be used by students (after it has been suitably scrubbed to make sure that 
privacy concerns are met). If the institutions are unwilling to make the information available, they may need 
to be encouraged to do so (by government agencies in the case of publicly-funded universities, by donors or 
accreditation boards in the case of private institutions). Some of this information is already publicly available 
(sometimes due to government regulations) either individually at the institutions, or collected by agencies or 
commercial entities (see figure 2, with information about American universities collected by US News and 
World Report). The Predictive Analytics Reporting Framework (PAR) project has shown that multiple 
universities can work together to unify and aggregate their data [7], Such work provides hope for 
implementing DIY analytics. 

Academic Life 

The student-faculty ratio at Kent State University is 21 :1 . and the 
school has 47 percent of its classes with fewer than 20 students. 

The most popular majors at Kent State University Include: Business. 

Management. Marketing, and Related Support Services: Health 
Professions and Related Programs: Communication. Journalism, 
and Related Programs: Education; and Psychology. The average 
freshman retention rate, an indicator of student satisfaction. Is 77.3 
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Figure 2. Information about an American institution of higher learning 
(http://colleges.usnews.rankingsandreviews.com/best-colleges/kent-state-university-3051) 

Further initial reflection on this research indicates that in order for the information to be relevant for 
decision-making for non-expert users (students) this use of visual analytics will be crucial. Furthermore, 
given the platform preferences of today’s students, the data presentation should be accessible from mobile 
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platforms (smartphones and tablets). This idea is reflected in our future prototype system in this area, which 
is described in the following section. 


3. CONCLUSIONS AND FUTURE RESEARCH 

This paper has described our work-in-progress in the area of DIY Analytics. We have identified several 
scenarios where the interests of institutions of higher education and their students diverge, leading to an 
opportunity to add value for students. We have also identified a possible problem in the implementation of 
this idea - the lack of ownership by the students of the data involved, though we hope to be able to overcome 
this problem in the short term by scraping publicly available data off of university websites (along with 
government agency and other organization sites) and in the medium and long term through a more open 
access to institutions’ data (scrubbed for privacy). Further, we have identified a few areas that DIY Analytics 
must address, based on the target audience. We continue to refine all of these ideas. 

We are currently in the initial stage of developing a prototype system in DIY Analytics. Our prototype 
will allow prospective students to explore various programs at multiple universities. The system will use SAS 
solutions for Hadoop [13]. SAS Visual Analytics and SAS Mobile BI will be used to produce an application 
accessible from mobile devices to meet the needs of today’s students. Interviews with current university 
students will be used as part of the design process of this prototype and it will be evaluated by experiments 
with a group of target users. These results will be reported in a future paper. 
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ABSTRACT 

This paper discusses some possibilities of gamification and remixing process for music education. Analyses also the 
concepts of gamification, mashup, remix and presents its possible usage in education - music teaching - through the 
development of the project/educational game “Flappy Crab”. 

The article begins with a brief introduction to the concepts of education, gamification, remix and mashup and software 
development. After that, we will make the summary presentation of the music edu-game “Flappy Crab”, a clone of the 
GEARS Studios Flappy Bird , developed for mobiles devices with the Unity3D game-engine. 


KEYWORDS 

Gamification, remix, mashup, software development, art education, awareness. 


1. INTRODUCTION 

In the twenty-first century, the consolidation of a new culture based on ubiquitous digital information, 
requires a review of practices in search to set a critical update on the educational landscape. Therefore, it is 
necessary to invest in new knowledge’s acquisition processes directly related with the new technologies, 
namely with recourse a strategies like the gamification or the remix, given that they are undeniably present as 
much in the educational processes (formal or not) as in the context of the cybernetic culture. 

This article aims to bring two major contributions, namely, the presentation of a prototype for a topic 
related with music educational game, “Flappy Crab”, a clone of the GEARS Studios Flappy Bird, developed 
for mobile platforms such as IPhone, Windows Phone (Smartphone) and tablets with the help of the Unity3D 
game-engine and. on another level, the study of all possibilities and advantages that creative pedagogical 
techniques such as the remix/mashup and gamification can have on the teaching of art-education and music. 

This paper is organized as follows: Section 2 discusses some of the implications that strategies like the 
gamification or the remix have in the education. In section 3 we will make the presentation of the edu-game 
“Flappy Crab”. Section 4 presents the conclusions relating to this stage of the development of the project. 


2. GAMIFICATION, REMIX AND MASHUP IN EDUCATION 
2.1 Gamification and Learning 

Gamification is a relatively new concept that has garnered considerable momentum over the last years (Lee & 
Hammer, 2011; Kapp, 2012; Deterding el al, 2011). It’s a strategy that aims applying the mechanics of 
gaming to non-game activities to change behaviours. At its root, the concept applies the mechanics of gaming 
to non-game activities to change people’s behaviour. When used on the educational field, gamification is the 
process of integrating game dynamics and game mechanics into learning activities and didactics objects such 
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as tests, quizzes, training exercises, edu-games, etc., in order to drive engagement, internal or intrinsic 
motivation and participation. 

In this context, we can define game mechanics as the set of rules and rewards that make up game play a 
satisfying and motivational activity, in another words, they are the aspects that make it challenging and 
educative, or whatever other emotion that the gamified activity hope to evoke. These emotions, in turn, are 
the result of desires and motivations that we could call game dynamics. The most common game mechanics 
(Lee & Hammer, 2011) include 

• Points: studies driven by the University of Chicago are showing that points are fantastic motivators 
and can be used to reward users/students across multiple levels or dimensions of the gamified 
activity. In fact, we know that people love to be rewarded and. when interacting with the point 
system, they feel like they’ve gained something. 

• Levels: are often defined as point thresholds, so the students (or users) can use it to indicate a higher 
status and control access to bonus content on the game. 

• Challenges, badges, and achievements, trophies: the introduction of goals in an activity make the 
students (users) feel like they’re working toward something. Normally, the challenges should be 
configured based on actions that we desire to improve, and rewarding users/students that accomplish 
reaching certain milestones with badges, achievements or trophies. 

• Leader boards or “high-score table”: in the context of gamification, high-score tables are used to 
track and display desired actions, using completion to drive valuable behaviour. In intrinsic 
motivation terms, they are one of the most important features of a game, bringing aspiration factor 
to the process. 

In our days, the education system recognize the power that the strategy of gamification has to improve 
students’ engagement, building processes of metacognition and the potential for solving a variety of 
problems related to the lake of intrinsic motivation, so important in music learning (Csikszentmihalyi, 1990; 
Wu, 2014). In fact, the four-factor model for instructional design, ARCS, developed by John Keller, is one of 
the best ways to study the degree of motivation that students achieve by de use of gamified activities ( Kapp, 
2012 ). 

In a very literal sense, the educative system has always used gamification in their processes by applying 
scores on assignments that can be considered points; and also, according this perspective, the graduation is a 
level achieved and a diploma is a form of granting a badge of confidence (Lee & Hammer, 2011). However, 
this game -based system doesn’t seem very engaging for the students; we think that perhaps the education 
process and particularly music learning could be improve by adding the game factor trough technology. 

2.2 Remix and Mashup 

The modern society is characterized by individuals who are no longer willing to be merely passive receptors 
of data contents but they want also produce, by cutting, sampling, paste or jam with content, in order to 
create something which is distinctive of their own social and creative innovation (Katz, 2009). The 
techniques of remix and mashup can be traced to the first experiences of great artists like Pablo Picasso but 
they gained importance and visibility with the rise of the internet and new digital technologies, which has 
made ease to re-use and remix the existing cultural contents. 

Mashup refers to the combination of contents or functionality from more than one external source to 
create a new entity and can be considered as the remix practices that use one or many materials, media or 
other sources for creating new artefacts trough alteration, re-combination, manipulation and copying. In 
doing so, the sources of origin may still be identifiable yet not perceived as the original version. The 
techniques of remix and mashup are extremely appreciated by digital natives, so, in the present time, they are 
extensively practice (Buzato, et al, 2013). 

In the music edu-game “Flappy Crab” we have used remix and mashup techniques regarding the game 
idea - cloned from an addictive mobile game - and the visual design, created from clipart images obtained 
from an online open source database. The characters in the game have been carefully selected to constitute a 
form of artwork of approach (the musical figures/quavers with human facies). These temporary use 
mechanisms aim to help the pupil to understand the relative space/time/time as well as make eye contact with 
the musical notation (McPherson & Gabrielsson, 2002). This educational application intends to motivate the 
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student to advance their learning in a playful way and may be used at any time and space and not necessarily 
in school context. 

Although the game prototype is still in a very early development stage, the concept was introduced to a 
set of eleven students aged between 9 and 12 years; these users have considered the gameplay excellent and 
the interface very intuitive. 


3. THE PROJECT “FLAPPY CRAB” 

The music educational game “Flappy Crab” aims to promote music learning based on the Csikszentmihalyi‘s 
Flow model that states “during optimal experience, a person is in such psychological state where he or she is 
so involved with the goal driven activity that nothing else seems to matter” (Csikszentmihalyi, 1990). The 
target audience for “Flappy Crab” places itself between the ten and the twelve years of age and attends the 2 nd 
Cycle of Basic Education; this edu-game focuses on the use of mobile computing devices, gamification and 
remixing to enhance student engagement and promote learning in music and art education. 

3.1 Game Mechanics and Storyboard 

The reason beneath the idea of gamifying one didactic object determines how we should develop the project. 
In the case of the game mechanics of the edu-game "Flappy Crab", designed to enhance informal learning of 
musical notation, we did include the seven design concept proposed by Priebatsch (2010), the follow: 

• The game includes visual measuring experience and progress by means of the score counter. 

• Rapid feedback provide through the progression dynamic, by means of sound warnings when the 
player gains or lose points. 

• Multiple short-term goals, that leads to a bonus level. 

• Rewards for effort and task completion, by the inclusion of the bonus level always the player 
anteing a certain sore (Wu, 2014), and 

• An element of uncertainty because the player never knows exactly what will be the bonus level form 
(Priebasch, 2014). 

As in the game that copies, "Flappy Crab" is the side-srolling mobile game featuring 2D graphics style 
based on a series of individual sprite sheets. The objective is to steer the flying red crab named Flappy, wich 
falls and moves continuously to the right, between each set of oncoming sargasso sea plants without colliding 
with them, which otherwise will restart the game over and over. In same time, fishs bonus with the music 
figure inside continuously moves towards the player on a diferent velocity rate. The crab briefly flys upward 
and in front each time the player taps the screen, so is an game easy to learn but hard to master ( Figure 1 ). 

The player is scored with twenty five points on the number of sargasso sea plants and fishs bonus the crab 
catches and passes trough: when achieve a certain score number, passes automatically to an bonus level. This 
level has one diferent mechanics, since training aims the capability of absolute picth in assotiation with 
melodic memory. Here, the player must replicate a given melody with the seven music sounds hidden in 
several buttons on the game interface. Once the task is done, the player returns to one level with the same 
mechanics that the previous and so on. 


4. CONCLUSION 

With the emergence of the internet and mobile games, gamification and mashup techniques are also 
becoming increasingly important in the educational system. Games are now achieving a new paradigm where 
their purpose is no longer for entertaining only but also for educating. In fact, games can offer interactive 
learning activities that can foster creativity and long term knowledge. Understanding how and why 
gamification works, in what contexts it is most effective, and what the limits are of this approach will be 
highly useful in sorting out the useful bits. The mashup/remix phenomenon is yet new and can be considered 
as a coevolving process of user-generated content media. In a society where the information flows are 
ubiquitous, the educational process, namely at the level of aesthetic expression, needs engaging the learners' 


259 



ISBN: 978-989-8533-23-4 ©2014IADIS 


awareness to the possibility of re-combining digital media available and make the respective re- 
interpretation, in a process where even the undesirable "copy and paste" method can morph into a creativity 
foundation. 

Drawing on the experience of gamification and mashup/remix techniques, this explorative paper makes 
the presentation of one prototype educational game, called “Flappy Crab”, that emulate de famous GEARs 
game and aims gamifying de learning process of music formal reading. 
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ABSTRACT 

The goal of this study is to explore the understanding and view of Malaysian Higher Education Institution (HEI) students 
on the concept of Massive Open Online Courses (MOOC). Questionnaires will be used to identify respondents’ 
knowledge (or lack of) of MOOC. Participants will then given a different set of questionnaire based of their knowledge of 
MOOC. Participants who have never heard of MOOC will receive a questionnaire that contains the general 
characteristics of a normal MOOC course, and will be asked to give feedback on a 6-point Likert Scale. By knowing the 
students understanding and view of MOOC, we will gain important information on the needs and concerns for MOOC 
courses in Malaysia. This short paper will describe an overview of the proposed study. 


KEYWORDS 

Massive Open Online Courses, online learning, higher education institutes, Malaysia. 


1. INTRODUCTION 

This study will explore the understanding and view of Malaysian Higher Education Institution (HEI) students 
on the concept of Massive Open Online Courses (MOOC). It is important as MOOC was only recently been 
discussed in this country, even though it gained a lot of popularity in the US in two years ago (Hu 2013). By 
knowing the students understanding and view of MOOC, we will gain important information on the needs 
and concerns for MOOC courses in this country. This study can also contribute as an unofficial advert, to 
promote MOOC in Malaysia. 


2. MOOC IN BRIEF 

MOOC is a free online program that has sparked a lot of interest in the US two years ago (Hu 2013). Though 
it has been around prior to 2011, but people only start taking notice of the program when some of the world 
elite universities such as Harvard, MIT, Stanford University and Princeton University decided to take part. It 
is accessible to a global audience, and it is free. It is also flexible as it allows you to study at your own pace 
and time. MOOC is describe as much more impersonal and effortless than its other online learning 
predecessor. It is open to anybody who has access to its online location; having thousands of globally located 
students for a single course; students grading each other’s work, and students teaching each other (Baggaley 
2013). The goal is to create a community of learners who bring personal resources and perspectives to those 
offered by the teacher and to embark on a journey of discovery that is personal, yet include other students 
(Downes 2011; Mackness, Mak & Williams 2010). 

However, most universities give no credits for MOOC courses (Palin 2013). For a fee, you may receive a 
certificate upon completion of a MOOC course, but not credits. One of the major concerns was the online 
identity of the student. But, now a few universities have introduced a new requirement where credits may be 
awarded if the students are able to demonstrate that they have met the course learning outcomes through prior 
learning assessments either via relevant assignment or an on-campus exam (Lee 2013; Parr 2013). 
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Another major concern with MOOC courses is the lack of instructional design (Baggaley 2013). It is all 
over the place, in blogs, discussion forums, tweets and videos. There is no one straight path that you can 
follow from start to finish. On top of that, as a student, you will be swamped by about 500++ postings, topics 
or questions to discuss daily, which came through from all the other students. Many of the course developer 
worries about the technical part of the course; overlooking the most critical part which is having good 
pedagogical approaches and proper course instructions (Mackness, Mak & Williams 2010). 


3. ONLINE LEARNING IN MALAYSIA 

Adopting new technology in teaching and learning has recently been a new “tagline” at public higher 
education institutions (public HEIs) in Malaysia. Online application enablers such as Padlet, Slideshare and 
Blendspace, were introduced to the academics, with high hopes that they will implement them in their 
teaching and learning activities. By blending the use of technology, it is hoped that learning in public HEIs 
will no longer mean sitting in the lecture room for two to three hours per subject per week. The aim is to 
promote self-learning and to enculturate lifelong learning among the student of HEIs. 

The Malaysian Education Minister has urged HEIs to be more proactive in adopting new technology for 
teaching and learning. The Education Ministry is aiming for 30% of courses at public HEIs to be offered 
online by 2015 (The Star 2013), which is in line with the blueprint on enculturation of lifelong learning for 
Malaysia 2011-2020 (Malaysian Education Ministry 2014). Lifelong learning is said to be the utmost 
important strategy for the nation’s human capital development. Though formal higher education in 
universities and colleges still plays a major role, lifelong learning can provide a second chance to those who 
missed their opportunity to pursue higher education before. It also offers the possibility to the working 
population to enhance their education level. The use of technology will be a vital supporting agent which will 
enable the students to study at their own time, at their own place and their own pace. 

Another important document is the National Higher Education Strategic Plan (PSPTN), which is a 
document that translates the direction of national higher education for the future that focuses on the 
development of quality human and intellectual capital (Mohamed Amin et al. 2011). The Education Ministry 
has developed 21 Critical Agenda Projects (CAPs) to ensure the implementation of PSPTN, and e-learning 
has been identified as one of the most important CAPs. 

In trying to implement online learning in Malaysia, Norazah, Mohamed Amin and Zaidan (2011) found 
out that for a student, the challenges that they faced in online education programme are usually due to lack of 
/ limited access to the Internet. Aside from that, lengthy response time from lecturers, lack of content, time 
consuming, lack of interesting content, uninteresting content and it is not like other social applications e.g., 
Facebook also contributed to student’s lack of interest to participate in online education programme. These 
findings show that HEIs need to upgrade and enhance their network infrastructure across all campus. The 
increase in broadband facilities is the utmost important factor for a functioning online learning environment. 

Besides that, to be able to participate in a MOOC program effectively, one must be ICT literate. Punie 
and Cabrera (2005) describes ICT literacy as not only having basic computer literacy, but rather a higher 
order skill such as: knowing where to search for certain information; how to process and evaluate 
information; how to assess the reliability and trustworthiness of websites and other online sources; and many 
others. Katz and Macklin (2007) argued that the problem faced by most tertiary level students today is their 
inability to navigate, evaluate and use the plethora of online information available today. This is in accord 
with Mat-jizat (2012) findings where 148 Malaysian trainee teachers were given a task-based ICT literacy 
test, and the result shows that the ICT domain that the trainee teachers find the hardest were to identify and 
judge credible websites. 

Aside from that, Malaysia’s strict rules and regulations of the Malaysian Qualifications Agency (MQA) 
might become a deterring factor. The inflexibility of standards and criteria developed to ensure equivalency 
of academic programmes in HEIs might not yet ready for MOOC. In Malaysia, a statutory body known as the 
Malaysian Qualifications Agency (MQA) is responsible for accrediting academic programmes provided by 
HEI in Malaysia (MQA 2013). MQA’s main role is to ensure the quality of education of HEIs in the country, 
by developing standards and criteria as national references for conferment of degrees. MQA also evaluates 
foreign's qualifications and assesses them for equivalency with Malaysian HEI qualifications (MQA 2013). 
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Aside from MQA, there is another body that is the Public Services Department (PSD) which is a 
government body that manages the public service human resources in Malaysia. Recognition of academic 
programmes from PSD is crucial as employment in the public sector will be based on academic programmes 
that are recognised by PSD only (Bahagian Pembangunan Modal Insan 2013). Usually, PSD will only 
recognise academic programmes that were accredited by MQA. 

In addition to employability in the public sector, students from an academic programme that are not 
accredited by MQA will face difficulties if they want to further their studies at any local public HEI; apply 
for credit transfers; or apply for sponsorships. Therefore, there is going to be a possibility that students who 
passed their MOOC courses with flying colours might have a problem in applying for jobs in the public 
sector or pursuing a higher degree. It somehow defeats the purpose of MOOC program where it is suppose to 
help student acquire knowledge at their own pace and space, and later use that knowledge to have a better 
chance in improving his own social, employment and economic status. 


4. CURRENT DEVELOPMENT FOR ONLINE LEARNING 

In May 2014, the Malaysian Ministry of Education held a MOOC Course Development Technical Committee 
Meeting in Putrajaya (Malaysian Education Ministry 2014). The aim of the meeting was to introduce the new 
OpenCourseWare2MOOC (OCW2MOOC) programme which is targeted to be implemented in September 
2014. This pilot programme is more OCW than MOOC. Only four courses will be offered: 1) Islamic and 
Asian Civilization; 2) Ethnic Relations; 3) ICT Competency and 4) Entrepreneurship. These four courses are 
compulsory courses that every HEI students in Malaysia must be enrolled. 

HEI students will have access to this online material and flipped teaching, or flipped classroom will be 
implemented in their respective lectures, based on the online material. Contents from the online material will 
also be included into the student’s assessment. 


5. THE STUDY 

The objectives of this study are; 

• to explore the level of understanding of MOOC programs among HEI students in Malaysia; and 

• to understand the HEI students view of MOOC. 

When investigating the feasibility of a MOOC program, many researchers focused on sources, 
communications, technologies, context and learning management systems (Cabiria 2012). However, for this 
study the students’ view of MOOC programs (second objectives of the study) will be investigated according 
to five categories: 

• program feasibility in Malaysia; 

• ICT literacy; 

• network structure; 

• credit transfer; and 

• Instructional design. 

The students will also be given an opportunity to suggest another category that they feel important to have 
for MOOC program to be successful and largely accepted and acknowledged in Malaysia, not only by its 
prospective online students, but also by other HEIs in Malaysia. 


6. METHODOLOGY 

This study will use a questionnaire developed based on the five identified categories. Validity of the 
questionnaire will be determined by a group of seven experts with relevant background in online / distance 
education and knowledge in MOOC. The reliability of the questionnaire will be determined through a pilot 
test using 30 random Malaysian HEI students. 
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The main study will be conducted on three top HEI in Malaysia (based on the country National Ranking). 
The research population will be all final year students from the three HEI. Two sets of questionnaires will be 
used to gather information / views depending on their familiarity with MOOC program. 


7. CONCLUSION 

This study will explore the understanding and view of Malaysia’s HEI student of MOOC programs. At the 
end of this study, the students’ view of the MOOC program will be summarised. It is hoped that these 
findings could benefit relevant government ministries in making decision about MOOC or online learning in 
Malaysia. The findings could also improve our understanding on the requirements and demands of MOOC in 
South East Asia. It is hoped that the favourable view gathered can provide useful information for MOOC 
program developer. 
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ABSTRACT 

In this study, the “Information Seeking Strategies” scale that was developed by Mills, Rnezek and Wakefield (2013) has 
been adapted to Turkish language. A confirmatory factor analysis was run and subsequently, two items were removed. 
The fact that these two items are related to learning in a traditional class environment may be interpreted in such a 
manner that the participants prefer online environments. As a result of the analysis of the items, it has been observed that 
ICT tools are employed for information seeking rather than information sharing. 
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1. INTRODUCTION 

As technology develops, the storage of and access to information is also transforming. Information, once 
accessed via printed material and then transferred onto digital media such as computers, CD-ROMs, and 
DVD-ROMs, has now become accessible anywhere and by everyone, owing to internet technologies. 

Researchers study the information and communication technologies (ICT) through various aspects. Its’ 
effects on learning in various contexts have been the subject of numerous studies. In this respect, researchers 
have also examined ICT learning on the basis of information searching and information sharing concepts. 
ICT is instantaneous and mobile providing equality of voice, access and communication in real time. Via ICT 
everyone has a voice and this causes informational convergence and pollution, intended or not (Kuhlthau, 
2010 ). 

Guided inquiry is important in this process in which information is structured. Guided Inquiry bring a 
different approach to teaching and learning information literacy (Kuhlthau, Maniotes, & Caspari, 2007). the 
main concepts of locating, evaluating and using information, which can be transferred to wide range of 
situations of information seeking and use, are learned by student (Kuhlthau, 2010). 

Guided inquiry is a planned process which progresses towards goals and is executed by means of inquiry. 
The Information Search Process (ISP) model explains the thoughts, actions and emotions in six stages. These 
stages are initiation, selection, exploration, formulation, collection, and presentation. Model of the ISP is 
presented in Figure 1 . 
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Figure 1. Model of information search process (Kuhlthau, 2004) 

Individuals perform active interpretation through the framework of their personal points of view. While 
this may not be the same for every individual, the information obtained from various sources resembles the 
individual’s present information. Formal information sources and informal ones acquired through 
experiences in daily life engage in continuous interaction (Kuhlthau, 1991). 

Particularly in relation to education, it is very important for students to ensure that the information they 
find is appropriate, reliable and relatively the most qualified, as well as to search and reach information on 
the internet which they use as the primary source of information for their homework, projects, and 
presentations. On the other hand, it is frequently stated that it is very hard for both students and instructors to 
find correct and reliable information on the web, and that the expected quality is seldom achieved (Mazman 
& A§kar, 2013). 

1.1 Aim of the Study 

ICTL scale was developed to allow research on how students choose to interact with ICT tools in relation to 
educational information seeking and sharing. In this study, the “Information Seeking Strategies” scale that 
was developed by Mills, Knezek and Wakefield (2013) has been adapted to the Turkish language and it is 
aimed to test how this scale, which was originally applied in the USA, works on a sample in Turkey. 


2. METHODOLOGY 

2.1 Data Collection Tool and Process 

The ICT Learning survey was developed by Mills, Knezek and Wakefield (2013) as part of a psychometrics 
course. The goal of the scale is to determine how students use ICT tools with respect to educational 
information seeking and sharing (Mills, Knezek & Wakefield, 2013). 

The scale contains 15 items and 2 factors, ‘Information Seeking’ and ‘Online Information Sharing’. The 
scale is a 5 Likert type, ranked from “1-Strongly disagree” through “5-Strongly agree”. 

Prior to translation into Turkish, the required permission was obtained from the authors.. It was then 
translated into Turkish language by 3 people. A form, which was developed on the basis of the translations, 
was then translated back into English by a linguist. Finally, the researchers determined the best expression for 
each item. The scale was delivered to students via e-mail and through social media. 

2.2 Participants 

A total of 148 participants from various universities and departments in Turkey took part in the study. The 
demographic data for the participants are listed in the Table 1 below. 
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Table 1. Demographic variables of participants. 



Variables 

Frequency 

Percent 


Female 

110 

76.9 

Gender 

Male 

33 

23.1 


18-25 

114 

79.7 

Age 

26-35 

25 

17.5 


36-40 

4 

2.8 


The numeric data regarding the participants’ intended use of the internet are given in Table 2. 


Table 2. Participants’ internet usage purposes 


Usage Purposes 

N 

Min 

Max 

X 

S 

For home works 

143 

1 

3 

2.14 

0.83 

For personally 

143 

1 

3 

2.40 

0.74 

For area of interests 

143 

1 

3 

2.47 

0.77 


As the analysis regarding intended use is examined, it is observed that participants use the internet 
primarily for their own areas of interest ( X = 2.47), then, for their personal studies ( X = 2.40) and finally for 
searching in connection with their homework and projects ( X = 2.14). 

2.3 Data Analysis 

SPSS 17.0 and LISREL 8.7 software were used to analyze the data. During the scale adaption process, 
confirmatory factor analysis (CFA) was employed with LISREL 8.7. 

Before CFA, the normal distribution of data was checked and the outliers were determined. Kaiser- 
Meyer-Olkin Measure of Sampling Adequacy (KMO) and Bartlett's Test of Sphericity tests were tested in 
order to determine whether the sample was adequate. The KMO and Barlett Test of Sphericity results are 
given in Table 3. 


Table 3. KMO and Barlett Test of Sphericity results 


Kaiser-Meyer-Olkin Measure of Sampling Adequacy. 

0.764 

Chi-Square 

469.241 

Bartlett's Test of Sphericity 

105 

Sig. 

.000 


Results of KMO and Bartlett's Test of Sphericity show that data has been fit for the analysis. 


3. FINDINGS 

The original scale has two dimensions:information seeking and online information sharing. Primarily, a first 
level confirmatory factor analysis was performed to test whether this structure is validated in our sample. The 
values obtained as a result of the analysis [x 2 (89, N=143) = 173.01, p<.000, RMSEA= 0.082, S-RMR= 
0.085, GFI= 0.86, AGFI= 0.81, CFI= 0.87, NNFI= 0.84, IFI= 0.87] indicated that the model did not fit well, 
in other words, did not confirm the original factorial structure. Thus, goodness of fit statistics and 
modification indexes were examined and it was decided that a number of items were to be removed from the 
scale. It was observed in the path diagram that the t value of items 9 (t= -1.31) and 12 (t=0.05) were not 
statistically significant. 
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The analysis was repeated for the remaining 13 items. According to this analysis, it was observed that ['// 
(64, N=143) = 141.91, p<.000, RMSEA= 0.093, S-RMR= 0.087, GFI= 0.87, AGFI= 0.81, CFI= 0.88, NNFI= 
0.85, IFI= 0.88] the model did not show a good fit. At this stage, the modification connections suggested in 
the analysis results were applied and the analysis was re-run. The modification connections were established 
for between Item 7 and Item 4, Item 10 and Item 8, Item 1 land Item 8, Item 1 land Item 10, Item 14 and Item 
10. Consequently, the values [x 2 (59, N=143) = 87.12, p<.000, RMSEA= 0.058, S-RMR= 0.064, GFI= 0.91, 
AGFI= 0.87, CFI= 0.95, NNFI= 0.94, IFI= 0.95] indicated that the model was in good fit. Table 4 compares 
the standard adaptability criteria (Schermelleh-Engel, Moosbrugger & Muller, 2003) and research results. 


Table 4. Comparison of standart goodness of fit indices and research results 


Fit Measure 

Good Fit 

Acceptable Fit 

Model Measures 

x2/df 

0< X 2/df <2 

2< x2/df <3 

1.48 

RMSEA 

0< RMSEA <0.05 

0.05< RMSEA <0.08 

0.058 

S-RMR 

0< S-RMR <0.05 

0.05< S-RMR <0.10 

0.064 

IFI 

0.95< IFI <1.00 

0.90< IFI <0.95 

0.95 

NNFI 

0.97< NNFI <1.00 

0.95< NNFI <0.97 

0.94 

CFI 

0.97< CFI <1.00 

0.95< CFI <0.97 

0.95 

GFI 

0.95< GFI <1.00 

0.90< GFI <0.95 

0.91 

AGFI 

0.90< AGFI <1.00 

0.85< AGFI <0.90 

0.87 


Connection diagram according to first level CFA (standardized solution) is presented in Figure 1. t-values 
connection diagram is also presented in Figure 2. 






Figure 1. Standardized Solution 


Figure 2. t values 


The Cronbach’s a value, the scale’s reliability coefficient, was found to be 0.79. This value indicates that 
the scale is reliable. The Cronbach’s a coefficients regarding the factors are given in Table 5. 


Table 5. Reliability test result 


Factors 

Cronbach a 

Information Seeking (Items 1, 4, 7, 8, 10, 14) 

0.62 

Information Sharing (Items 2, 3, 5, 6, 11, 13, 15) 

0.71 
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4. CONCLUSION 

ICT has become an element that affects our learning and working in the fields of education, economy, 
politics and society. New skills, new information and new ways of learning have become prominent within 
these dynamic environments. Inability to adapt to these environments leads to complications, disappointment, 
and possible failure. ICT teaching has gained importance in order to ensure creativity and interpretation. 

The 15 item ICTL scale developed by Mills, Knezek and Wakefield (2013) was applied in a sample from 
Turkey and finalized as a 13 item form to be used in Turkish setting. The scale’s validation and reliability 
were tested and subsequently, two items were removed. The fact that these two items are related to learning 
in a traditional class environment may be interpreted in such a manner that the participants have a preference 
for online environments. As a result of the analysis of the items, it has been observed that ICT tools are 
employed for information seeking rather than information sharing. 
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ABSTRACT 

This project seminar aims at creating and evaluating a manual for interdisciplinary projects as part of a learning process. 
Working together, pedagogies and students from different disciplines assess tools and recommendations for successful 
collaborations while developing an app for the cathedral in Freiberg. 

As part of the project the students gain expertise in project management and are introduced to techniques form other 
disciplines which complement their work for the app. The results of the project seminar and the manual may be assigned 
to other interdisciplinary projects. 


KEYWORDS 

Team Project based Learning, Team Teaching, Cross-disciplinary Learning, Evaluation, Research Seminar. Higher 
Education 


1. PROJECT BASED LEARNING AS LEARNING APPROACH 

A major aspect related to a research and presentation of Cultural Heritage objects is their visualisation for the 
public. Thus media are mostly created within projects by interdisciplinary teams (Munster, 2013). Moreover, 
a project’s success and efficiency highly relies on experience related to both, an application of professional 
knowledge and interdisciplinary cooperation skills. 

While these scenarios are increasingly relevant for a practical work for example in museums, they 
currently play just a minor role in academic education of historians or media developers. A well introduced 
approach to acquire competencies to deal with complex and multi -perspective problems via Project based 
learning (PBL) as “learning by doing”, originally introduced by Dewey (Dewey and Dewey, 1915). By 
definition project based learning is a “student-centered approach in which students determine what they need 
to learn” in order to solve nontrivial, real-world problems and where “teachers act as facilitators” to foster 
both problem solving and learning progress (Barrows, 2002). In comparison to other learning designs PBL 
promises a greater satisfaction and level of engagement of students as well as a higher level of long-term 
retention of acquired competencies (SUobel and van Barneveld, 2009). 

While performed by student work groups as teams during project based learning skills like a cooperative 
ability, critical reasoning, creative thinking, responsibility, and communication are promoted (Moursund. 
2003, Lee and Lim, 2012). PBL is particularly effective when combined with computer technology and 
introduces students especially from humanities to new tools (Chang and Lee, 2010). Potential problems result 
for example from a non-equal contribution of team mates to the final outcome and a fair negotiation of 
individual conUibutions as well as a high management load and a lack of transparency for teachers about 
project group activities and social constellations (Lee and Lim, 2012). 


2. THE PROJECT 

Based on the assumption that skills for interdisciplinary project work which will be highly useful when 
entering the job market are best acquired within a practical project (Strobel and van Barneveld, 2009), a 
guide for such projects might encourage more teachers to use them as a practice related educational sUategy. 


270 



1 1th International Conference on Cognition and Exploratory Learning in Digital Age (CELDA 2014) 


The project seminar is a student project where an app is created within an educational project with the 
outcome of a manual for interdisciplinary student projects. Supported by the S axon Centre for Teaching and 
Learning in Higher Education , this project seminar is cooperation between the Institute of Art and Music 
Studies and the Media Center of the TU Dresden as well as the Chair for German Literature and Medieval 
Studies of the TU Chemnitz. The project started in April of 2014 with 30 student participants at both bachelor 
and master level. A final presentation of the results was set for October 2014. Students involved in the project 
belong to the fields of art history, linguistic and geo science. The students of the humanities research aspects 
of architecture within the cathedral and its purpose for communication with the community. Visualisations 
like reconstructions and animations by the geo based students emphasise the results. 

The development and observation of the project seminar as a collaboration between students and 
pedagogies of a variety of disciplines ensures the practical relevance as well as the transferability concerning 
other interdisciplinary projects. Reflections and explication of methods and results during the seminar are 
fundamental to ensure the success of the project and the manual as a final outcome. 

2.1 Learning Conception 

A general learning objective of the project is to support a development of several competencies of learners. 
While team project based learning on multimedia addresses complex tasks a learning conception and related 
learning objectives base on clues from various explanation models. 

2.1.1 Gain Experience 

While knowledge in academic learning mostly focuses on explicit and theoretical knowledge, its practical 
application in complex scenarios is seldom part of academic education. According to a problem solving 
process a solution quality highly depends on prior real or imagined experiences of individuals for a certain 
situation (Tsoukas, 2006) and the availability of extant patterns for a solution (Simon, 1972). Moreover, these 
schemes highly depend on disciplinary prerequisites and a professional qualification (Goodwin, 1994) and 
are fundamental for Cognitive Load theory (Collins et al., 1987, Hatcher et al., 1996) as an explanation 
model for the cognitive processes of learning. Implications for our project are to set learning objectives for 
gaining practical expertise on application of academic knowledge, performing cross-disciplinary teamwork 
and applying gained expertise to a certain use case consecutively. Another main challenge for the students is 
to define an original task, a level of support and required outcomes. 

2.1.2 Development of Teamwork- related Competencies 

Each part of the cathedral was handled by a single cross-disciplinary student team (1-5 members). Following 
implications from cognitive load theory during the early stages of cooperation a more intensive guidance and 
supervision of these groups as well as a predefined schedule are required to reduce cognitive workload 
affected - beside the former described acquisition of personal competencies - by team-building processes and 
organisational learning (Wegner, 1986, Argyris and Schon, 1978). Moreover, as suggested by project 
management a self-organisation of teams has to be monitored continuously (Turner, 1999). Due to these 
reasons research assistants ensure the dissemination of information, questions and progress between the 
teachers and student teams. A competent assistant should be available for each discipline involved or 
required. The high workload of management and monitoring tasks during the project seminar calls for the 
involvement of assistants (Dunlap, 2005). 

2.1.3 Enhance Abilities for Interdisciplinary Communication 

Practice based competencies and experience are closely related to implicit knowledge (Polanyi, 1966). A 
major challenge for team work and interdisciplinary cooperation is the expression and explication of 
knowledge (Schon, 1983). Another closely related challenge of knowledge transfer processes in cross- 
disciplinary projects results from differing disciplinary terminologies. While more sophisticated code patterns 
like language require a synchronisation of individual interpretation and understanding (Wilkes-Gibbs and 
Clark, 1992) architectural structures as natural (Tversky, 2002), in sense of “everyday experienced” visual 
media are highly suitable to build a common ground for interdisciplinary communication (Munster et al.). 
Architecture can support a communication of more abstract and sophisticated concepts like a construction 
order as a boundary object (Star and Griesemer, 1989). 
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2.1.4 Enable Competencies to Create Knowledge Media 

A transfer of individual and team mental models into multimedia content is challenging in many ways. As 
highlighted for visual and textual media by Mintzberg and Westley (Mintzberg and Westley, 2010) mental 
processing for a creation of media highly differs and different media are suitable for different information. 
While the targeted application can contain various media, competencies for selecting, processing and 
combining of media need to be developed. Another issue is related to the adjustment of the proposed 
application for visitors. Due to this process a role switching takes place and student work groups work as 
designers of learning media proposed to others, anticipating user needs for usability and user experience and 
instructional design. 

2.2 Layout of the Project 

Based on implications for structuring a practice -based learning process (Collins et al., 1987) as far as 
processes of the working world, the seminar follows the typical layout of a project. A basic timetable sets 
deadlines for different tasks within the project seminar. As the seminars components dedicated to knowledge 
acquisition and educational instructions, the courses and study trips are designed to fit the projects timetable. 
Each of the three cooperating partners offers a course for knowledge building in their field of expertise as 
well as skills in project management. Previously acquired theoretical knowledge is channelled and adapted in 
order to help solve the assignment. The students heavily influence the courses by pointing out current issues 
and needs. The final result of the students work is an app for smartphones which provides information on the 
cathedrals architecture for its visitors. The content and its visualisations for the cathedral app are carried out 
by teams with students from different fields of study. The student-student interaction fosters and demands an 
active participation and engagement of each student within the team (Lee and Lim, 2012). With the help of 
acquired project management skills the students plan and supervise the project’s tasks and timetable within 
the group. Developing a common ground and language, taking on responsibility, supervising and motivating 
each other are challenges that need to be overcome. The seminar introduces the students to technology- 
supported PBL (communication tools, research tools, scaffolding tools, project management tools, tele 
collaboration tools) as well as multimedia technology-assisted PBL (production tools that enable students to 
organize and present their work through multimedia) (Chang and Lee, 2010). Every student takes over the 
role as mentor for their own discipline while identifying possibilities other disciplines may provide. 
Additionally to working on the app, the students are also involved in the creating of the guide for 
interdisciplinary projects through their feedback. Alternating periods of project work and reflection about the 
proceedings and their results form a recurring pattern. Four study trips to the cathedral in Freiberg and a 
public presentation of the app are intended. Every trip focusses on certain tasks or project steps like the 
introduction of the teams and the research objects, exploration of the object, data acquisition and presentation 
of the results. During the trips the teachers engage in team teaching in order to encourage multiple 
perspectives, promote dialogue, increase participation and improve evaluation and feedback (Anderson and 
Speck, 1998). By definition team teaching involves two or more teachers sharing teaching expertise and 
engaging in reflective dialogue with each other (Chang and Lee, 2010). The “interactive nature of team 
teaching may be a potential source of intellectual stimulation and cognitive development for learners as well 
as faculty” (Hatcher et al., 1996). 

2.3 Evaluation and Feedback 

A concept for formative evaluation of the seminar was adapted to process and specify the perception of 
challenges and goals, develop and exam certain provisions and determine the quality and suitability of the 
teaching concept (Kromrey, 2001). In order to gain insight into the success and acceptance of the seminar, 
the study trips are accompanied by evaluations to provide “frequent opportunities for formative assessment 
by both students and teachers” (Barron et al., 1998). Due to the recurring trips, the evaluations are carried out 
iteratively and in loops. A loop consisted of: perception of challenges, acquisition of information, adjustment 
of the seminar. Therefore, questionnaires and group discussions are used to ask questions, collect answers, 
inform about, discuss and evaluate solutions and progress. The group discussions function as qualitative 
interviews where subjective opinions and individual experiences are taken into account as well as general 
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directions for all participants and an explorative approach of issues may be distinguished. Comments and 
feedback by all participants will be considered and turned into recommendations for interdisciplinary student 
projects relying on team project based learning. A coordination committee consisting of teachers and 
research assistants collects and prepares the project’s results for the adjustment of courses as well as the 
publication of a manual for interdisciplinary student projects. 


3. PRELIMINARY IMPLICATIONS 

After currently three of four study trips including questionnaires, group discussions and additional feedback 
some preliminary implications can be drawn. The projects progress concerning the app as well as the manual 
has been positively reviewed by all participants. A special appeal is provided by the concept of a research 
project carried out with a variety of disciplines involved. Thanks to the high degree of individual supervision 
by the teachers and assistants, the general dedication to the project exceeds the dedication in common 
seminars by far. The seminar has shown that working in teams is still a novelty to students from humanities. 
Because of study routines and the focus on research they are used to working alone. Dividing the workload 
and relying on team mates proved to be a challenge, but the experiences during the project changed their 
attitude towards team projects positively. Taking over as mentors of their discipline builds up their 
confidence within the group and helps to face and solve problems collectively. A few recommendations 
based on student’s feedback were already identified. The students note the importance of advertising the 
project correctly. It needs to be emphasized that it is a research project which differs greatly from common 
projects concerning the approached and the expected results. Uncertainties about the involvement of all 
disciplines regarding a time budget and assignments caused reluctance amongst some students. Suggestions 
for coping with the concern are to keep the involvement of every discipline transparent and include it in the 
advertising of the seminar. The students agree on the need to have a course where all participants attend apart 
from the study trips. The course may focus on project management and help the teams set up a basis for 
communication, time management and a group hierarchy as opposed to dealing with it during the other 
courses separately. While setting up the concept of the seminar the engagement of the students was 
underestimated. Advice, tutorials and instructions prepared by students for each other may be promoted and 
demanded. The app for the cathedrals has proved to be a very nice incentive for the students. Seeing that their 
work and effort will be put to further use and be appreciated by visitors has increased the student’s 
motivation and builds pressure to excel. The publicly accessibly app may also be a reference for future job 
applications. 


4. CONCLUSION 

Focusing on the learning scenario by a perspective of problem solving “making a picture of problems 
require[s] deeper integration of ideas” and a more intense coordination between team members (Mintzberg 
and Westley, 2010). Even if visual communication strategies would foster to build a common ground for 
interdisciplinary communication (Munster et al., 2014) and promote long term retention of learning 
outcomes, other learning scenarios may ease an entry and accessibility for students. The experiences and 
knowledge acquired during the project equips the students with a variety of essential skills for a career entry. 
However, seminar projects require a high degree of supervision and much time to be carried out (Blumenfeld 
et al., 1991, Barron et al., 1998). Coordination by e.g. research assistants is just as necessary as a well 
prepared layout of the project. Moreover, it is necessary to focus “on questions or problems that "drive" 
students to encounter 11 (Thomas, 2000). Featuring collective purpose as the final result of the project is a 
very effective way to motivate students and help overcome difficulties by working together. 
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ABSTRACT 

This research examines the relationship between career related self-concept and dimensions of informal learning of 
science. The overlapping dimensions of career interest development and informal learning suggest that self-directed 
informal learning of science can advance individual self-concept for possible scientific self. Possible selves and future 
scientific selves theories are presented as a perspective for understanding career related aspirations, goals, and fear. In 
this preliminary research the author seeks to examine the possible role that informal science learning may play in 
students’ sense of scientific self by examining connections between wanting to work in science and dimensions of 
informal learning and career interest development. Findings from a pilot study of future science selves among n = 63 
students in grade 8 are discussed. Additional research is planned among students groups who visit a laser interferometer 
gravitational wave observatory (LIGO) with hand-on science exploratorium in the Southwest region of the United States. 


KEYWORDS 

Possible selves, future scientific selves, the career interest development process, informal learning, school science 


1. INTRODUCTION 

The Government Accountability Office (GAO) of the United States reported in 2010 that approximately three 
billion federal dollars were spent on science, technology, engineering, and mathematics (STEM) education 
programs (GAO, 2012). However, research has yet to identify a set of best practices that will increase 
students’ participation in STEM professions (Nixon, Meikle, & Borman, 2013). The informal-to-formal 
learning of science is thought to be key to encouraging participation in science -related careers (Osborne & 
Dillon, 2007). Possible selves are recognized as a gauge of career -related self-perceptions. For the 
manifestation of possible science selves, informal learning and general possible selves within the career 
development process share dimensions of self-motivation, as well as social and cultural contexts. The shared 
dimensions of informal learning and possible selves would suggest that the self-directed informal learning of 
science would advance individual self-concepts for the development of possible scientific selves and lead to 
increased interest in participation in science -related professions. This pilot study and the larger study that is 
introduced in this paper will examine the potential role of informal science learning experience in the 
development of students’ sense of scientific self. 


2. CONCEPTUAL RATIONALE 

Many of the dimensions of informal learning have counterparts within the theories of career interest 
development, warranting an examination of the connection between informal learning and career interest 
development. Rennie, Feher, Dierking, and Falk (2003) examined the National Association for Research in 
Science Teaching (NARST) 1999 committee policy statements on informal science education to present key 
facets of informal learning that can guide research. These research domains include: 1) self-motivated 
dynamics, 2) situational contexts, 3) sociocultural facets, 4) cumulative nature, and 5) cognitive components. 

Career interest development for self-concept has overlapping dimensions with the five key facets of 
informal learning presented by NARST. It is, by nature, self-motivated within the maturation process as 
self-image comes to define who an individual is and what that person would like to become (Erikson, 1950). 
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Career learning experiences are authentically situated in apprenticeships, mentoring, and explorations. Career 
self-concept is influenced by personal experience in social, cultural, and cognitive realms (Merolla & Serpe, 
2013). Finally, career awareness, interest, and self-concept are dynamic and cumulative in nature (Ginzberg, 
1972). 

2.1 Learning and Career Interest 

All learning, including informal learning, is a cumulative process involving learning experiences and various 
reinforcements that take place over time (Rennie et al., 2003). Learning and career awareness are strongly 
mediated by social interaction, conversation, and experience (Rennie et al., 2003) in a process of socio- 
cognitive development. Ginzberg (1972) defined occupational choice theory during his extensive study of 
vocational theories in the 1950s as consisting of three developmental periods: fantasy (before age 11), 
tentative (between ages 11 and 17), and realistic (between age 17 and young adulthood). Within occupational 
choice theory, career choice awareness develops through exploration, role-play, and experience in real-world 
contexts. The fantasy years are important to the career interest development process because students are 
most receptive to possible career options during this phase. The subsequent tentative and realistic phases are 
associated with a narrowing of career interest. This narrowing of career interest may be the same 
phenomenon that is depicted as a developmental loss of possible selves by Oyserman and Fryberg (2006). 
Urajnik, Garg, Kauppi, and Lewko (2007) recognized that experiential influences of context and experience, 
beyond the personal characteristics of adolescents, can be models for the prediction of scientific career 
choice. 

2.2 Possible Scientific Self 

Possible selves — self-images of current identity and future personas (Erikson, 1950) — can provide a measure 
of self-concept for career interest. Possible selves are examples of mental components of adolescent 
development that support the solidification of person and formation of occupational identity (Super, 1980). 
Within Markus and Nurius’ (1986) possible selves theory, career interest can be viewed as an expression of 
self-perceptions and self-conceptualized selves. Based on psychological theories advanced by William James 
during the years 1890-1950, possible selves can be viewed as the images of self that are desired, hoped for, or 
feared (Oyserman & Fryberg, 2006). These images of self are thought to work together to motivate behavior 
(Shepard & Marshall, 1999). Possible future science selves (Packard & Nguyen, 2003) may prove to assist in 
understanding science -related career interests during the career interest development process. As future self- 
views, possible science selves are yet to be verified within a social context and are therefore thought to be 
impressionable to change and influenced by environment (Markus & Nurius, 1986). 


3. MODES OF INQUIRY 

This pilot study of relationships between aspects of science career -related self-concept was based on a quasi- 
experimental design for collection of quantitative data. The search design was limited by constraints of social 
science research within a public school. Data analysis was conducted recognizing that consideration must be 
given to confounding extraneous variables that can jeopardize internal validity with quasi -experimental 
design. Data analysis did not focus on the identification of causal relationships due to possible influence of 
extraneous variables, such as “history, maturation, testing, instrumentation, regression, or selection” 
(Campbell & Stanley, 1963, p. 8) that would restrict the validity of claims for cause and effect. 


4. DATA SOURCES 

Data for this pilot study was originally collected in the fall of 2013 for a study seeking to identify predictors 
of STEM career interest among middle school students (Mills, 2014). Participants were 8th grade students 
from two schools in the same school district. These participants consisted of 30 boys and 33 girls. Some 
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students, 45 of the 63, had been selected by lottery to attend a STEM magnet high school the following 
school year. 

Future science selves were gauged with new subscales of the Career Interest Questionnaire, (CIQ), 
(Bowdich, 2009). The CIQ subscales gauged perceptions for future scientific selves, the attitudes of 
parents/society to scientific work, and perceptions of the work of scientists. These subscales have internal 
consistency and acceptable to very good reliabilities, as interpreted by DeVellis (1991), for the subjects of 
this study. See Table 1. 


Table 1. Career Interest Questionnaire (CIQ) Subscales 


Scale/Subscale 

Item Number 

Cronbach’s Alpha 

(Interpreted by DeVellis) 

CIQ/ Future scientific self 

5,6,7,8 

.91 Very good 

CIQ/ Attitudes of parents/society 

2,4,9 

.80 Very good 

CIQ/ Perceptions of the work of scientists 

3,10,11 

.70 Respectable 

CIQ “I would like to have a career in 
science” single item. 

1 

- 


The SSS subscales were analyzed for students’ preferences for science, technology, engineering, and 
mathematics. 

The following research questions were examined: 

• Research Question #1 will be the following statement: “/ would like to have a career in science. ” be 
significantly related to the following: 

• Future scientific self (CIQ) 

• Attitudes of parents and society (CIQ) 

• Perceptions of the work of scientists (CIQ) 

• Attitude towards science, technology, engineering, or mathematics (Stem Semantic Survey (SSS), 
(Tyler-Wood, Knezek, & Christensen, 2010)) 

• Research Question #2: Will student’s future scientific self differ by gender? 

• Research Question #3: Will eighth grade students who have been selected by lottery to attend a STEM 
high school have a significantly different perception of self-concept for future scientific self! 


5. PRELIMINARY FINDINGS 

Preliminary findings for the three research questions are as follows: 

• Research Question #1. Bivariate correlation analysis was examined for trends between the statement “7 
would like to have a career in science. ” and the following CIQ subscales: 

• Future scientific self (CIQ), a significant correlation was identified (r = .732, p < 000) 

• Attitudes of parents and society (CIQ) was significantly and positively related to the desire to have 
a scientific career ( r = .692, p < 000) 

• Perceptions of the work of scientists (CIQ) was significantly and positively related to the desire to 
have a scientific career (r = .705, p < 000) 

• Only attitude towards science was significantly and positively related to the desire to have a career 
in science, (r = .211, p = 028) 

• Research Question #2. Analysis of variance revealed no significant difference in perceptions of CIQ 
future scientific self by gender. Eighth grade boys were found to have a mean response of 3.48, while 
girls had a mean response of 3.59 (ascending scale of 1 to 5) of CIQ future scientific self. 

• Research Question #3. Analysis of variance did not reveal a significant difference in perceptions of CIQ 
future scientific self between students selected by application/lottery to attend a STEM magnet high 
school. Students selected for the STEM school were found to have a mean response of 3.59, while 
students who would attend regular neighborhood schools had a mean response of 3.39 (ascending scale of 
1 to 5) on the CIQ subscale, future scientific self. The only significant difference measured was for 
perceptions of the work of scientists (p = 0.41) subscale. Students who applied and were selected by 
lottery for the STEM magnet school were more positive regarding personal enjoyment and the social 
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value of doing scientific work, with a mean response of 3.93; while students bound for regular 
neighborhood schools had a mean response of 3.46 (on an ascending scale of 1 to 5). 

These early findings indicate that the desire to have a future scientific career is strongly related to socially 
and culturally mediated factors that are associated with informal learning and the career interest development 
process. It is interesting to note that the girls in this study did not differ significantly from the boys. 


6. SIGNIFICANCE AND FUTURE RESEARCH 

Informal learning is self-motivated, voluntary, and driven by choice and interests — such learning is 
dependent on authentic contexts and physical setting (Rennie et al., 2003). The career interest development 
process and career -related self-concept share dimensions with informal learning. An understanding of 
scientific self-concept and informal learning experiences may prove to be important to efforts aimed at 
increasing interest in the learning and liking of science and subsequent participation in science-related 
careers. For example, scientific stereotypes, thought to result from gender socialization, affect the career 
aspirations of adolescents. “If the caricature of the scientific personality and lifestyle does not mesh with the 
student’s interests, beliefs, and values, she or he is unlikely to become committed to being a scientist” 
(Etzkowitz, Kemelgor, & Uzzi., 2000, p. 47). Informal science learning opportunities that allow observation 
of professionals at work can help to dispel stereotypes, which influence career motivation. Informal learning 
of science may also be key to providing the motivation to stay connected to science as student attitudes 
towards school science decline across the school years (Yager, 1996) while students are thought to maintain 
separate, more positive attitudes towards real-world science (Osborne, Simon, & Collins, 2003; Ebenezer & 
Zoller, 1993). 

Additional research is planned to examine students’ attitudes to informal learning and career interests for 
future scientific self. The pilot study presented here indicates that there are shared dimensions of career 
interest development, such as possible science selves, and informal learning experiences. A broader study is 
underway that will examine pre-post data from participants who visit a laser interferometer gravitational 
wave observatory (LIGO) that houses an education center with hands-on science exploratorium at an 
experimental physics research facility. Effects of the LIGO informal science learning experience on future 
scientific self will be examined. 
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ABSTRACT 

This research, based on the case study of edX at Seoul National University, which is running Korea's first Massive Open 
Online Courses (MOOCs), discussed and proposed the roles of principal facilitators, the process, and the relationships 
among various facilitators in selecting, designing, opening and administrating MOOCs classes. Researches on MOOCs so 
far have been deficient in practical consideration for actual MOOCs administration system and process demanded by 
universities, teachers, class designers and administrators who wish to develop and run MOOCs as well as activities and 
strategies that needed to be implemented step-by-step by each member that comprises the system. In particular, since 
most researches on MOOCs as well as establishment of MOOCs organizations have been done in the United States and 
Europe, there is a need for researches done based on actual case studies of MOOCs implemented in Korea in order to 
facilitate more effective MOOCs administration under different environments, e..g, a Korean-type MOOCs. In 
conclusion, principal facilitators for MOOCs administration largely consists of MOOCs administration, MOOCs support 
and teachers. Each facilitator, based on online/face-to-face form of training and communication, engages in agreement, 
design, development and administration of MOOCs programs. Each process is defined through repetition of activities and 
strategies as well as feedback. Using the process and activities needed for designing and administrating MOOCs in 
practice proposed by this research, it is hoped that there will be further researches on designing and administrating 
MOOCs more effectively under a variety of environments. 


KEYWORDS 

Massive Open Online Courses (MOOCs), MOOCs Model for Managing 


1. INTRODUCTION 

Massive Open Online Courses (MOOCs) are open online courses operated by leading universities and 
educational organizations for people around the world. Through MOOCs, interactive learning takes place 
between teacher and student/system and student not only by using learning resources such as videos and 
documents but also learning activities such as solving practice questions, doing assignment and engaging in 
discussions. (Alario-Hoyos et. al., 2013) By offering high quality educational contents and support for 
teaching/learning, MOOCs aims to achieve the goal of providing retraining opportunities and sharing of 
knowledge from the perspective of lifetime education and enhanced teaching/learning experience at school 
from the perspective of supporting higher education. As a result, the program has been rapidly expanding 
since 2012 via Coursera(http://www.coursera.org), edX( http://www.edx.org) and 
Udacity( http://www.udacity.com) as an educational model (Martin, 2012) as well as a business model 
(Daniel, 2012) 

There are researches being done on the definition and the significance of MOOCs (Clara & Barbera, 2013; 
Mackness, Waite, Roberts, & Lovegrove, 2013; Yuan & Powell, 2013), course design and evaluation (Cross, 
2013; Meyer & Zhu, 2013) and analysis of student characteristics and learning pattern (Breslow et al., 2013; 
Milligan, Littlejohn, & Margaryan, 2013) based on potential and influence of MOOCs. However, on the other 
hand, the researches and case studies of the schools who want to run MOOCs, teachers of MOOCs classes, 
system of operating MOOCs needed by course designer and administrator, strategic plans and actions 
required for those who make up the system are lacking. In particular, since establishing MOOCs 
organizations and related researches have been done primarily in the United States and Europe, it is 
necessary to conduct researches based the examples of MOOCs implemented in Korea in order to 
administrate MOOCs programs more effectively under different learning environment. 
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In this research, based on the case of Seoul National University edX (SNUx), is proposed methods of 
selecting, designing and introducing principal facilitators for classes and the procedure for actually 
administrating the class; and the roles and relationship among various facilitators. 


2. RESEARCH METHODS 

This research covers the activities done and the materials produced from May 13, 2013 when Seoul National 
University began preparing for agreement with edX, the organization that manages MOOCs to April 30, 
2014 when the school has 1 class open and running, has designed and developed additional 2 classes and is in 
the process of developing a sequel to the class that is currently running. 

First, people or organizations needed to select, design, develop and administrate MOOCs were chosen as 
the principal facilitator and, based on the agreement made between Seoul National University and edX, 
activities to be performed by each facilitator has been listed and categorized. Second, things to be considered 
during design, development and administration of MOOCs have been derived based on the analysis on 
MOOCs related documents, well -performing classes within edX and information on class administration 
provided by MOOCs administrating organization. Finally, programs that were not examined in the previous 
two steps were additionally derived during the process of actually selecting, designing and developing 
MOOCs. 


3. RESEARCH RESULTS 

Principal facilitators for MOOCs administration can be divided into MOOCs administration team, MOOCs 
support team and MOOCs instructors. Each facilitator is to perform actions in the course of agreement, 
design and development, and training and communication pertaining to MOOCs. In this research, MOOCs 
administration team is edX and MOOCs support team is SNUx of Seoul National University registered in 
edX, consisting of Center for Teaching and Learning (CTL) and Task Force Team (TFT). Finally, MOOCs 
instructors are faculty and teaching assistants who are either running or will be running a class in SNUx. 
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Figure 1. Principal facilitators for MOOCs administration, steps of actions to be taken, and accompanying activities 
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The picture above is a diagram of each facilitator, the steps of action and the activities pertaining to each 
facilitator and action steps. The following is the survey of major activities that take place among the 
facilitators in each step. 

3.1 Agreement 

Most MOOCs administration, excluding Udemy( http://www.udemy.com), currently have agreements with 
MOOCs support organizations such as universities, educational organizations and corporations throughout 
the world for more effective and higher quality management. Agreements are largely made between MOOCs 
administrating organization and MOOCs support organization such as universities, educational organizations 
and corporations throughout the world; and between MOOCs supporting organization and MOOCs teachers. 

Agreements between MOOCs administrating organizations and support organizations usually go through 
discussions to reach basic agreement on infrastructure and administration methods as well as intellectual 
property rights, schedule of courses, etc. (edX, 2013a) Then, a MOOCs support organization selects classes 
through administration committee and works with teachers of the selected classes on various issues such 
administration schedule, designing and administration methods of learning activities, allocation of design and 
development tasks, selection of teaching assistants, whether to issue certifications and, ultimately, discusses 
specifically how to use MOOCs to achieve innovation in university education. 

3.2 Designs and Development 

Things that are needed to be developed in order to administrate MOOCs are largely promotion, overall 
structure of classes, videos and subtitles and learning activities. First, for promotion, information about class 
that includes the class title, photos and resume of faculty and teaching assistant, class descriptions, a class 
schedule and the required hours of study per week is made available in public eight weeks before the 
scheduled start of class. Also, e-mails are sent to students who applied for the class two weeks before the 
start of class to announce the start of class to remind them of their interest in classes so that they do not miss 
the class. 

Considering the students' concentration level and attendance rate, the length of the class is set for 5-8 
weeks and class objectives, the name of learning activities and materials and their count and sequence are 
designed and developed according to a weekly schedule. For SNUx, it used to consist of 13 weeks worth of 
videos and was divided into 2-3 classes. Next, videos, important learning materials, were designed and 
developed, and uploaded under the corresponding title. Although videos should ideally be divided to be 3-6 
minutes video each (edX, 2013b), one of the classes in SNUx was 20-30 minutes in length due to the nature 
of the subject and was used without dividing it further. Learning activities such as practice problems, 
discussions, questions, assignments and tests are developed using relevant web-based authoring tools and 
uploaded under the corresponding title with links to related web sites. In particular, when certifications are 
issued, there has to be a verification process for a multiple number of students to verify their participation in 
learning process. Therefore, a class must be designed so that difficulty level and the frequency of the class is 
maintained at an appropriate level within the boundary of not lowering the student motivation for learning. 

3.3 Administration 

One of the more important characteristics of MOOCs is that it is run to allow students to conduct an effective 
learning process instead of just designing and developing learning materials and programs. (Clara & Barbera, 
2013; Milligan, Littlejohn, & Margaryan, 2013). MOOCs is made available to a multiple number of students 
and therefore technical approaches, e.g., learning system and authoring tools, are taken to facilitate effective 
and efficient administration. Flowever, in addition to such technical support, faculty and teaching assistants 
must encourage students to conduct self-led studying and continuous participation by answering their 
question, encouraging student discussions, selecting outstanding responses as well as using e-mails, 
announcement and videos, etc. 
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3.4 Training and Communication 

In order for agreement, design and development and administration of MOOCs to take place in effective 
ways, it is important to provide training in MOOCs authoring tools, learning platform, teaching plan and 
promotion. Also, in addition periodic training, use of e-mails and communication via video conference on a 
regular basis is crucial for success of a class and in maintaining its quality. Furthermore, by creating a 
consultive group among MOOCs support organizations and holding academic symposiums hosted by 
MOOCs administration committees, MOOCs is being transformed and developed into a locus of recognizing 
the need for new directions and strategies for university education and sharing of relevant information. 
(Coursera, 2014: edX, 2013b) 


4. DISCUSSION AND CONCLUSION 

This research, based on the actual case study of MOOCs administration at Seoul National University, 
proposed how to select, design, develop and administrate MOOCs classes and principal facilitators the 
system, the process required and the roles and relationships among facilitators. It is expected that, with 
accumulation of more knowledge in administrating MOOCs classes in various academic disciplines, this 
research will be used as a basis to further researches on developing more effective MOOCs design and 
administration under more diverse environments. Going further, it is hoped that this research will become a 
foundation for developing a Korean-type MOOCs that incorporate unique linguistic characteristics and 
educational environment as well as existing educational infrastructure, thus leading a way for more effective 
administration of MOOCs. 
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ABSTRACT 

This paper examines literature on the development of self-knowledge for possible selves — how an individual thinks about 
oneself and one’s potential future selves (Markus & Nurius, 1986). Future science selves research, a recent offshoot of 
possible selves theories, centers on the development and loss of future possible scientific selves and factors such as 
academics, apprenticeships, or serious science gaming that may influence the science self and science career interest. 
Science career persistence is thought to be related to scientific exposure, reflection in real-world contexts, mentoring 
programs, and intensive math and science academic programs. While serious science gaming may prove to encourage 
science selves by providing positive virtual science-related experiences, there is limited research on best learning contexts 
or long-range effects of academic science gaming on adolescents. 
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1. INTRODUCTION 

Expansion of the science- and math-related workforce to meet rising demand requires attention to theories 
highlighting the career interest development during the developmental years. The possible selves metaphor of 
self-knowledge that examines an individual’s self-view has been described as the link between self-concept 
and motivation (Markus & Nurius, 1986). In general, self-concept research revealed the vast complexity of 
self-knowing (Gergen, 1972; Greenwald & Pratkanis, 1984), while possible selves added theories on 
self-concept to explain the role of self-images in personal functioning motivation for behavior and change 
(Markus & Nurius, 1986). Future science selves research examines the development and loss of future 
possible selves. This paper examines literature on learning and self-views such as possible selves, possible 
science selves, serious science gaming, and experiential learning that can guide efforts to expand 
participation in science- and mathematics-related occupations. It is hoped that such an examination will 
provide insights for educators, researchers, and instructional designers in their efforts to create and provide 
experiential and digital contexts that will help students develop future science selves. 


2. CONCEPTUAL RATIONALE 

Mental self-images can be viewed as self-perspectives of who an individual is at present, and who or what 
that person would like to become in the future (Erikson, 1950) or as components of adolescent development 
that support the solidification of person and formation of occupational identity (Super, 1980). The future 
selves theory regards career interest as one facet of individual development that is related to self-perceptions 
and imagined selves. Markus and Nurius (1986) developed future selves concepts based on the possible 
selves psychological theories advanced by American philosopher and psychologist, William James during the 
years 1890-1950. Possible selves have been summarized as the possible images of self that are desired or 
expected and negative images of the self that are to be avoided. These images of self are thought to work 
together to motivate behavior (Shepard & Marshall, 1999). Oyserman, Bybee, and Terry (2006) posited that 
self-image can impel action to achieve goals. Human developmental includes expansion and narrowing of the 
range of possible selves and accompanying career interest (Ginzberg, 1972). Oyserman and Fryberg (2006) 
found evidence to indicate a loss of possible academic selves during transitions from elementary school to 
middle school to high school. 
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2.1 Possible Science Selves 

Packard and Nguyen (2003) applied the lens of the possible selves to examine persistence in the science 
careers of women (n = 41) who were 18 to 21 years of age. They examined career plans at the end of high 
school and during the first and second years of college and also studied changing career trajectories. Thirty of 
the forty-one women began with math- and science-related career goals. Women who maintained initial 
interest or changed to a science-related career field shared common factors: strong interest, participation in 
academic and workplace opportunities, and supportive relationships with parents, friends, and mentors. 
Participants credited their persistence to awareness of science careers, reflection in real-world contexts, 
mentoring programs, and intensive math/science programs. 

2.2 Developing Science Selves through Serious Games 

Beier, Miller, and Wang (2012) examined how students “for whom there is neither mentor nor positive 
STEM [science, technology, engineering, and mathematics] -related experiences” acquire an interest in 
STEM. They postulated that it is important to create positive experiences for students early in the school 
years, before and during the middle school years, during the more impressionable years of human 
development. They report enthusiasm for forensics among middle school students who role-played a related 
video game at school and discussed the experience via social media. 

Foster (2008) suggested that video gaming may be effective in creating meaningful learning experiences. 
Lee and Hoadley (2007) indicated that massively multiplayer online games (MMOGs) can serve as valuable 
learning explorations that reinforce identity formation. Wonica (n.d.) proposed that avatar identities allow 
positive change among children via exploration in virtual worlds with an idealized self. Wonica (n.d.) 
believed that gaming identities can assist in identity formation for academic and emotional growth. Dodge, et 
al. (2008) examined video gaming as a bridge to cognitive and communicative experience for personal 
growth and transformation. Research along these lines is seeking to identify and harness the unique 
affordances of virtual worlds and role-play. Yet, the variety of options for learning in digital contexts poses 
many unanswered questions regarding the value of experience and the context of learning. 

2.3 Experiential Learning 

Experiential learning theories, which provide a comprehensive framework for learning contexts, based on 
works of Dewey, Lewin, and Piaget can guide in the identification of experiences and learning for science 
selves. Experiential learning emphasizes the importance of interaction of four types of abilities: concrete 
experience, reflective observation, abstract conceptualization abilities, and active experimentation abilities 
(Kolb, 1984). Higher forms of individual adaptation, such as the development of person and creativity, 
require conflict among these elements of individual ability and real-world experience in interactions between 
individuals, objects, and persons. Self-concept for possible selves is an example of individual adaptation and 
development of person. Research is needed to determine if serious science gaming, situated in virtual worlds, 
will prove to support aspects of experiential learning for the positive development of person. 


3. SYNTHESIS 

While the early research on the relationships of serious games, role-playing, and virtual environments to self- 
concept seems to support the notion that gaming experience may provide students with scientific role-play 
and support the development of possible selves among adolescents, experiential learning concepts indicate 
that individual development in complex life domains requires real-world interactions and negotiation 
between perceptions and concrete realities. Advances in digital technologies and the variety and quality of 
experiences that can be provided in virtual environments warrant an examination of the effectiveness of 
game-based exploration and apprenticeships among various age groups. While serious science gaming may 
prove to support the development of serious scientific selves and aspects of experiential learning, such as 
abstract conceptualizations and active experimentation, additional research is needed on the relationship of 
gaming to the development of self-concept and persistent possible science selves during the developmental 
years. 
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ABSTRACT 

This paper presents a novel classification scheme for Collaborative Intelligent Tutoring Systems (CITS), an emergent 
research field. The three emergent classifications of CITS are unstructured, semi -structured, and fully structured. While 
all three types of CITS offer opportunities to improve student learning gains, the full extent to which these gains may be 
achieved is subject to future research. 
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1. INTRODUCTION 

The purpose of this paper is to present a novel classification scheme used for Collaborative Intelligent 
Tutoring Systems (CITS). Intelligent Tutoring Systems mark the forefront of research in applied artificial 
intelligence to education. The field of Computer Supported Collaborative Learning focuses on how students 
learn in collaborative settings and how technology can enhance collaborative peer interaction and work 
(Dillenbourg and Crivelli, 2011; Nkambou et al., 2010). Recently, research efforts have focused on merging 
the affordances of both industries to capitalize on the benefits of group learning and adaptive support 
(Nkambou et al., 2010). The relatively new area of research necessitates a method of uniform evaluation that 
emphasizes main features from both ITS and CSCL literature. 


2. CITS CLASSIFICATION 

A comparative analysis between selected works within CITS literature resulted in a pioneering classification 
scheme that blends ITS and CSCL paradigms. The classification scheme is intended to support the analysis 
of design principles used within CITS. I propose that CITS can be classified and analyzed via the following 
dimensions: Modeling (how learners are modelled and whether the learning domain includes collaborative 
behavior). Group Dynamics (how groups and roles are determined). Collaboration Cues (the impetus of the 
initial collaboration and the timing of ongoing communication). Pedagogical Guidance (how the topic of 
communication is determined and how feedback and activity facilitation is implemented). Technology (the 
system tools for interaction). 

This classification scheme encompasses the main operational dimensions of CITS. CITS are a 
collaborative learning approach that uses technological tools (technology) to provide systematic support to 
learners (pedagogical guidance) working within groups (group dynamics) either assigned or at-will 
(collaboration cues) and maintained in computational representations as they work to achieve a learning goal 
(modelling). The scheme consists of three categories: unstructured, semi-structured, and fully structured. 
Unstructured CITS leave decisions regarding the collaboration to the user while fully structured CITS guide 
the entirety of the collaborative process. Semi-structured CITS intermingle user directed collaboration with 
system guidance. 
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Table 1 . Detailed CITS classification scheme which includes criteria for the five dimensions. 


CITS Classification Scheme Unstructured(U), Semi-Structured(S), Fully Structured(F) 

U 

s 

F 

Modelling (Target Audience & Objective) 




Provides individual support 

>/ 

>/ 

>/ 

Uses individual and/or group models to provide collaborative support 


>/ 

>/ 

Support concerns domain learning 

>/ 

>/ 

>/ 

Support concerns collaborative behavior 


>/ 

>/ 

Group Dynamics 




Users determine groups 

>/ 

>/ 


System or system requirements determine collaborative groups 



>/ 

Users determine roles 

>/ 

>/ 


System determines collaborative roles 



>/ 

Collaboration Cues (Impetus & Timing) 




Initial collaboration occurs at-will 

>/ 

>/ 


Initial collaboration encouraged for task 


>/ 


Initial collaboration required for task 


>/ 

>/ 

Collaborators determine when to communicate 

>/ 

>/ 

>/ 

System prompts collaborators to communicate 


>/ 

>/ 

Pedagogical Guidance 




Users determine activities 

>/ 

>/ 

>/ 

System determines activities 


>/ 

>/ 

Users determine how to collaborate and provide support to others without system guidance 

>/ 

>/ 


System guides collaboration and collaborators in how to provide support to others 


>/ 

>/ 

Technology 




Users determine what tools are best for collaboration and communication 

>/ 

>/ 


System restricts collaborators to a set of tools for collaboration and communication 


>/ 

>/ 

Distributed Learning Support 




Collaboration dependent on physical location of users 

>/ 



Collaboration independent of physical location of users 

>/ 

>/ 

>/ 


3. CONCLUSION 

The CITS framework shows a plethora of learning designs and capabilities exists that accommodate 
collaboration and adaptive support for education. The collaboration may be unstructured, semi -structured, or 
fully structured. The CITS framework provides a uniform tool for analysis of a variety of CITS while 
emphasizing main features required from both ITS and CSCL research. 
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